ceph/doc/start/hardware-recommendations.rst

   1 ==========================
   2  Hardware Recommendations
   3 ==========================
   4
   5 Ceph was designed to run on commodity hardware, which makes building and
   6 maintaining petabyte-scale data clusters economically feasible.
   7 When planning out your cluster hardware, you will need to balance a number
   8 of considerations, including failure domains and potential performance
   9 issues. Hardware planning should include distributing Ceph daemons and
  10 other processes that use Ceph across many hosts. Generally, we recommend
  11 running Ceph daemons of a specific type on a host configured for that type
  12 of daemon. We recommend using other hosts for processes that utilize your
  13 data cluster (e.g., OpenStack, CloudStack, etc).
  14
  15
  16 .. tip:: Check out the Ceph blog too. Articles like `Ceph Write Throughput 1`_,
  17    `Ceph Write Throughput 2`_, `Argonaut v. Bobtail Performance Preview`_,
  18    `Bobtail Performance - I/O Scheduler Comparison`_ and others are an
  19    excellent source of information.
  20
  21
  22 CPU
  23 ===
  24
  25 Ceph metadata servers dynamically redistribute their load, which is CPU
  26 intensive. So your metadata servers should have significant processing power
  27 (e.g., quad core or better CPUs). Ceph OSDs run the :term:`RADOS` service, calculate
  28 data placement with :term:`CRUSH`, replicate data, and maintain their own copy of the
  29 cluster map. Therefore, OSDs should have a reasonable amount of processing power
  30 (e.g., dual core processors). Monitors simply maintain a master copy of the
  31 cluster map, so they are not CPU intensive. You must also consider whether the
  32 host machine will run CPU-intensive processes in addition to Ceph daemons. For
  33 example, if your hosts will run computing VMs (e.g., OpenStack Nova), you will
  34 need to ensure that these other processes leave sufficient processing power for
  35 Ceph daemons. We recommend running additional CPU-intensive processes on
  36 separate hosts.
  37
  38
  39 RAM
  40 ===
  41
  42 Generally, more RAM is better.
  43
  44 Monitors and managers (ceph-mon and ceph-mgr)
  45 ---------------------------------------------
  46
  47 Monitor and manager daemon memory usage generally scales with the size of the
  48 cluster.  For small clusters, 1-2 GB is generally sufficient.  For
  49 large clusters, you should provide more (5-10 GB).  You may also want
  50 to consider tuning settings like ``mon_osd_cache_size`` or
  51 ``rocksdb_cache_size``.
  52
  53 Metadata servers (ceph-mds)
  54 ---------------------------
  55
  56 The metadata daemon memory utilization depends on how much memory its cache is
  57 configured to consume.  We recommend 1 GB as a minimum for most systems.  See
  58 ``mds_cache_memory``.
  59
  60 OSDs (ceph-osd)
  61 ---------------
  62
  63 By default, OSDs that use the BlueStore backend require 3-5 GB of RAM.  You can
  64 adjust the amount of memory the OSD consumes with the ``osd_memory_target`` configuration option when BlueStore is in use.  When using the legacy FileStore backend, the operating system page cache is used for caching data, so no tuning is normally needed, and the OSD memory consumption is generally related to the number of PGs per daemon in the system.
  65
  66
  67 Data Storage
  68 ============
  69
  70 Plan your data storage configuration carefully. There are significant cost and
  71 performance tradeoffs to consider when planning for data storage. Simultaneous
  72 OS operations, and simultaneous request for read and write operations from
  73 multiple daemons against a single drive can slow performance considerably.
  74
  75 .. important:: Since Ceph has to write all data to the journal before it can
  76    send an ACK (for XFS at least), having the journal and OSD
  77    performance in balance is really important!
  78
  79
  80 Hard Disk Drives
  81 ----------------
  82
  83 OSDs should have plenty of hard disk drive space for object data. We recommend a
  84 minimum hard disk drive size of 1 terabyte. Consider the cost-per-gigabyte
  85 advantage of larger disks. We recommend dividing the price of the hard disk
  86 drive by the number of gigabytes to arrive at a cost per gigabyte, because
  87 larger drives may have a significant impact on the cost-per-gigabyte. For
  88 example, a 1 terabyte hard disk priced at $75.00 has a cost of $0.07 per
  89 gigabyte (i.e., $75 / 1024 = 0.0732). By contrast, a 3 terabyte hard disk priced
  90 at $150.00 has a cost of $0.05 per gigabyte (i.e., $150 / 3072 = 0.0488). In the
  91 foregoing example, using the 1 terabyte disks would generally increase the cost
  92 per gigabyte by 40%--rendering your cluster substantially less cost efficient.
  93 Also, the larger the storage drive capacity, the more memory per Ceph OSD Daemon
  94 you will need, especially during rebalancing, backfilling and recovery. A
  95 general rule of thumb is ~1GB of RAM for 1TB of storage space.
  96
  97 .. tip:: Running multiple OSDs on a single disk--irrespective of partitions--is
  98    **NOT** a good idea.
  99
 100 .. tip:: Running an OSD and a monitor or a metadata server on a single
 101    disk--irrespective of partitions--is **NOT** a good idea either.
 102
 103 Storage drives are subject to limitations on seek time, access time, read and
 104 write times, as well as total throughput. These physical limitations affect
 105 overall system performance--especially during recovery. We recommend using a
 106 dedicated drive for the operating system and software, and one drive for each
 107 Ceph OSD Daemon you run on the host. Most "slow OSD" issues arise due to running
 108 an operating system, multiple OSDs, and/or multiple journals on the same drive.
 109 Since the cost of troubleshooting performance issues on a small cluster likely
 110 exceeds the cost of the extra disk drives, you can accelerate your cluster
 111 design planning by avoiding the temptation to overtax the OSD storage drives.
 112
 113 You may run multiple Ceph OSD Daemons per hard disk drive, but this will likely
 114 lead to resource contention and diminish the overall throughput. You may store a
 115 journal and object data on the same drive, but this may increase the time it
 116 takes to journal a write and ACK to the client. Ceph must write to the journal
 117 before it can ACK the write.
 118
 119 Ceph best practices dictate that you should run operating systems, OSD data and
 120 OSD journals on separate drives.
 121
 122
 123 Solid State Drives
 124 ------------------
 125
 126 One opportunity for performance improvement is to use solid-state drives (SSDs)
 127 to reduce random access time and read latency while accelerating throughput.
 128 SSDs often cost more than 10x as much per gigabyte when compared to a hard disk
 129 drive, but SSDs often exhibit access times that are at least 100x faster than a
 130 hard disk drive.
 131
 132 SSDs do not have moving mechanical parts so they are not necessarily subject to
 133 the same types of limitations as hard disk drives. SSDs do have significant
 134 limitations though. When evaluating SSDs, it is important to consider the
 135 performance of sequential reads and writes. An SSD that has 400MB/s sequential
 136 write throughput may have much better performance than an SSD with 120MB/s of
 137 sequential write throughput when storing multiple journals for multiple OSDs.
 138
 139 .. important:: We recommend exploring the use of SSDs to improve performance.
 140    However, before making a significant investment in SSDs, we **strongly
 141    recommend** both reviewing the performance metrics of an SSD and testing the
 142    SSD in a test configuration to gauge performance.
 143
 144 Since SSDs have no moving mechanical parts, it makes sense to use them in the
 145 areas of Ceph that do not use a lot of storage space (e.g., journals).
 146 Relatively inexpensive SSDs may appeal to your sense of economy. Use caution.
 147 Acceptable IOPS are not enough when selecting an SSD for use with Ceph. There
 148 are a few important performance considerations for journals and SSDs:
 149
 150 - **Write-intensive semantics:** Journaling involves write-intensive semantics,
 151   so you should ensure that the SSD you choose to deploy will perform equal to
 152   or better than a hard disk drive when writing data. Inexpensive SSDs may
 153   introduce write latency even as they accelerate access time, because
 154   sometimes high performance hard drives can write as fast or faster than
 155   some of the more economical SSDs available on the market!
 156
 157 - **Sequential Writes:** When you store multiple journals on an SSD you must
 158   consider the sequential write limitations of the SSD too, since they may be
 159   handling requests to write to multiple OSD journals simultaneously.
 160
 161 - **Partition Alignment:** A common problem with SSD performance is that
 162   people like to partition drives as a best practice, but they often overlook
 163   proper partition alignment with SSDs, which can cause SSDs to transfer data
 164   much more slowly. Ensure that SSD partitions are properly aligned.
 165
 166 While SSDs are cost prohibitive for object storage, OSDs may see a significant
 167 performance improvement by storing an OSD's journal on an SSD and the OSD's
 168 object data on a separate hard disk drive. The ``osd journal`` configuration
 169 setting defaults to ``/var/lib/ceph/osd/$cluster-$id/journal``. You can mount
 170 this path to an SSD or to an SSD partition so that it is not merely a file on
 171 the same disk as the object data.
 172
 173 One way Ceph accelerates CephFS filesystem performance is to segregate the
 174 storage of CephFS metadata from the storage of the CephFS file contents. Ceph
 175 provides a default ``metadata`` pool for CephFS metadata. You will never have to
 176 create a pool for CephFS metadata, but you can create a CRUSH map hierarchy for
 177 your CephFS metadata pool that points only to a host's SSD storage media. See
 178 `Mapping Pools to Different Types of OSDs`_ for details.
 179
 180
 181 Controllers
 182 -----------
 183
 184 Disk controllers also have a significant impact on write throughput. Carefully,
 185 consider your selection of disk controllers to ensure that they do not create
 186 a performance bottleneck.
 187
 188 .. tip:: The Ceph blog is often an excellent source of information on Ceph
 189    performance issues. See `Ceph Write Throughput 1`_ and `Ceph Write
 190    Throughput 2`_ for additional details.
 191
 192
 193 Additional Considerations
 194 -------------------------
 195
 196 You may run multiple OSDs per host, but you should ensure that the sum of the
 197 total throughput of your OSD hard disks doesn't exceed the network bandwidth
 198 required to service a client's need to read or write data. You should also
 199 consider what percentage of the overall data the cluster stores on each host. If
 200 the percentage on a particular host is large and the host fails, it can lead to
 201 problems such as exceeding the ``full ratio``,  which causes Ceph to halt
 202 operations as a safety precaution that prevents data loss.
 203
 204 When you run multiple OSDs per host, you also need to ensure that the kernel
 205 is up to date. See `OS Recommendations`_ for notes on ``glibc`` and
 206 ``syncfs(2)`` to ensure that your hardware performs as expected when running
 207 multiple OSDs per host.
 208
 209 Hosts with high numbers of OSDs (e.g., > 20) may spawn a lot of threads,
 210 especially during recovery and rebalancing. Many Linux kernels default to
 211 a relatively small maximum number of threads (e.g., 32k). If you encounter
 212 problems starting up OSDs on hosts with a high number of OSDs, consider
 213 setting ``kernel.pid_max`` to a higher number of threads. The theoretical
 214 maximum is 4,194,303 threads. For example, you could add the following to
 215 the ``/etc/sysctl.conf`` file::
 216
 217         kernel.pid_max = 4194303
 218
 219
 220 Networks
 221 ========
 222
 223 We recommend that each host have at least two 1Gbps network interface
 224 controllers (NICs). Since most commodity hard disk drives have a throughput of
 225 approximately 100MB/second, your NICs should be able to handle the traffic for
 226 the OSD disks on your host. We recommend a minimum of two NICs to account for a
 227 public (front-side) network and a cluster (back-side) network. A cluster network
 228 (preferably not connected to the internet) handles the additional load for data
 229 replication and helps stop denial of service attacks that prevent the cluster
 230 from achieving ``active + clean`` states for placement groups as OSDs replicate
 231 data across the cluster. Consider starting with a 10Gbps network in your racks.
 232 Replicating 1TB of data across a 1Gbps network takes 3 hours, and 3TBs (a
 233 typical drive configuration) takes 9 hours. By contrast, with a 10Gbps network,
 234 the  replication times would be 20 minutes and 1 hour respectively. In a
 235 petabyte-scale cluster, failure of an OSD disk should be an expectation, not an
 236 exception. System administrators will appreciate PGs recovering from a
 237 ``degraded`` state to an ``active + clean`` state as rapidly as possible, with
 238 price / performance tradeoffs taken into consideration. Additionally, some
 239 deployment tools  (e.g., Dell's Crowbar) deploy with five different networks,
 240 but employ VLANs to make hardware and network cabling more manageable. VLANs
 241 using 802.1q protocol require VLAN-capable NICs and Switches. The added hardware
 242 expense may be offset by the operational cost savings for network setup and
 243 maintenance. When using VLANs to handle VM traffic between the cluster
 244 and compute stacks (e.g., OpenStack, CloudStack, etc.), it is also worth
 245 considering using 10G Ethernet. Top-of-rack routers for each network also need
 246 to be able to communicate with spine routers that have even faster
 247 throughput--e.g.,  40Gbps to 100Gbps.
 248
 249 Your server hardware should have a Baseboard Management Controller (BMC).
 250 Administration and deployment tools may also use BMCs extensively, so consider
 251 the cost/benefit tradeoff of an out-of-band network for administration.
 252 Hypervisor SSH access, VM image uploads, OS image installs, management sockets,
 253 etc. can impose significant loads on a network.  Running three networks may seem
 254 like overkill, but each traffic path represents a potential capacity, throughput
 255 and/or performance bottleneck that you should carefully consider before
 256 deploying a large scale data cluster.
 257
 258
 259 Failure Domains
 260 ===============
 261
 262 A failure domain is any failure that prevents access to one or more OSDs. That
 263 could be a stopped daemon on a host; a hard disk failure,  an OS crash, a
 264 malfunctioning NIC, a failed power supply, a network outage, a power outage, and
 265 so forth. When planning out your hardware needs, you must balance the
 266 temptation to reduce costs by placing too many responsibilities into too few
 267 failure domains, and the added costs of isolating every potential failure
 268 domain.
 269
 270
 271 Minimum Hardware Recommendations
 272 ================================
 273
 274 Ceph can run on inexpensive commodity hardware. Small production clusters
 275 and development clusters can run successfully with modest hardware.
 276
 277 +--------------+----------------+-----------------------------------------+
 278 |  Process     | Criteria       | Minimum Recommended                     |
 279 +==============+================+=========================================+
 280 | ``ceph-osd`` | Processor      | - 1x 64-bit AMD-64                      |
 281 |              |                | - 1x 32-bit ARM dual-core or better     |
 282 |              +----------------+-----------------------------------------+
 283 |              | RAM            |  ~1GB for 1TB of storage per daemon     |
 284 |              +----------------+-----------------------------------------+
 285 |              | Volume Storage |  1x storage drive per daemon            |
 286 |              +----------------+-----------------------------------------+
 287 |              | Journal        |  1x SSD partition per daemon (optional) |
 288 |              +----------------+-----------------------------------------+
 289 |              | Network        |  2x 1GB Ethernet NICs                   |
 290 +--------------+----------------+-----------------------------------------+
 291 | ``ceph-mon`` | Processor      | - 1x 64-bit AMD-64                      |
 292 |              |                | - 1x 32-bit ARM dual-core or better     |
 293 |              +----------------+-----------------------------------------+
 294 |              | RAM            |  1 GB per daemon                        |
 295 |              +----------------+-----------------------------------------+
 296 |              | Disk Space     |  10 GB per daemon                       |
 297 |              +----------------+-----------------------------------------+
 298 |              | Network        |  2x 1GB Ethernet NICs                   |
 299 +--------------+----------------+-----------------------------------------+
 300 | ``ceph-mds`` | Processor      | - 1x 64-bit AMD-64 quad-core            |
 301 |              |                | - 1x 32-bit ARM quad-core               |
 302 |              +----------------+-----------------------------------------+
 303 |              | RAM            |  1 GB minimum per daemon                |
 304 |              +----------------+-----------------------------------------+
 305 |              | Disk Space     |  1 MB per daemon                        |
 306 |              +----------------+-----------------------------------------+
 307 |              | Network        |  2x 1GB Ethernet NICs                   |
 308 +--------------+----------------+-----------------------------------------+
 309
 310 .. tip:: If you are running an OSD with a single disk, create a
 311    partition for your volume storage that is separate from the partition
 312    containing the OS. Generally, we recommend separate disks for the
 313    OS and the volume storage.
 314
 315
 316 Production Cluster Examples
 317 ===========================
 318
 319 Production clusters for petabyte scale data storage may also use commodity
 320 hardware, but should have considerably more memory, processing power and data
 321 storage to account for heavy traffic loads.
 322
 323 Dell Example
 324 ------------
 325
 326 A recent (2012) Ceph cluster project is using two fairly robust hardware
 327 configurations for Ceph OSDs, and a lighter configuration for monitors.
 328
 329 +----------------+----------------+------------------------------------+
 330 |  Configuration | Criteria       | Minimum Recommended                |
 331 +================+================+====================================+
 332 | Dell PE R510   | Processor      |  2x 64-bit quad-core Xeon CPUs     |
 333 |                +----------------+------------------------------------+
 334 |                | RAM            |  16 GB                             |
 335 |                +----------------+------------------------------------+
 336 |                | Volume Storage |  8x 2TB drives. 1 OS, 7 Storage    |
 337 |                +----------------+------------------------------------+
 338 |                | Client Network |  2x 1GB Ethernet NICs              |
 339 |                +----------------+------------------------------------+
 340 |                | OSD Network    |  2x 1GB Ethernet NICs              |
 341 |                +----------------+------------------------------------+
 342 |                | Mgmt. Network  |  2x 1GB Ethernet NICs              |
 343 +----------------+----------------+------------------------------------+
 344 | Dell PE R515   | Processor      |  1x hex-core Opteron CPU           |
 345 |                +----------------+------------------------------------+
 346 |                | RAM            |  16 GB                             |
 347 |                +----------------+------------------------------------+
 348 |                | Volume Storage |  12x 3TB drives. Storage           |
 349 |                +----------------+------------------------------------+
 350 |                | OS Storage     |  1x 500GB drive. Operating System. |
 351 |                +----------------+------------------------------------+
 352 |                | Client Network |  2x 1GB Ethernet NICs              |
 353 |                +----------------+------------------------------------+
 354 |                | OSD Network    |  2x 1GB Ethernet NICs              |
 355 |                +----------------+------------------------------------+
 356 |                | Mgmt. Network  |  2x 1GB Ethernet NICs              |
 357 +----------------+----------------+------------------------------------+
 358
 359
 360
 361
 362
 363 .. _Ceph Write Throughput 1: http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/
 364 .. _Ceph Write Throughput 2: http://ceph.com/community/ceph-performance-part-2-write-throughput-without-ssd-journals/
 365 .. _Argonaut v. Bobtail Performance Preview: http://ceph.com/uncategorized/argonaut-vs-bobtail-performance-preview/
 366 .. _Bobtail Performance - I/O Scheduler Comparison: http://ceph.com/community/ceph-bobtail-performance-io-scheduler-comparison/
 367 .. _Mapping Pools to Different Types of OSDs: ../../rados/operations/crush-map#placing-different-pools-on-different-osds
 368 .. _OS Recommendations: ../os-recommendations