Deploy Hyper-Converged Ceph Cluster
===================================
:pve-toplevel:
+
+Introduction
+------------
endif::manvolnum[]
[thumbnail="screenshot/gui-ceph-status-dashboard.png"]
- Snapshot support
- Self healing
- Scalable to the exabyte level
+- Provides block, file system, and object storage
- Setup pools with different performance and redundancy characteristics
- Data is replicated, making it fault tolerant
- Runs on commodity hardware
- No need for hardware RAID controllers
- Open source
-For small to medium-sized deployments, it is possible to install a Ceph server for
-RADOS Block Devices (RBD) directly on your {pve} cluster nodes (see
-xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]). Recent
-hardware has a lot of CPU power and RAM, so running storage services
-and VMs on the same node is possible.
+For small to medium-sized deployments, it is possible to install a Ceph server
+for using RADOS Block Devices (RBD) or CephFS directly on your {pve} cluster
+nodes (see xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
+Recent hardware has a lot of CPU power and RAM, so running storage services and
+virtual guests on the same node is possible.
+
+To simplify management, {pve} provides you native integration to install and
+manage {ceph} services on {pve} nodes either via the built-in web interface, or
+using the 'pveceph' command line tool.
+
-To simplify management, we provide 'pveceph' - a tool for installing and
-managing {ceph} services on {pve} nodes.
+Terminology
+-----------
+// TODO: extend and also describe basic architecture here.
.Ceph consists of multiple Daemons, for use as an RBD storage:
-- Ceph Monitor (ceph-mon)
-- Ceph Manager (ceph-mgr)
-- Ceph OSD (ceph-osd; Object Storage Daemon)
+- Ceph Monitor (ceph-mon, or MON)
+- Ceph Manager (ceph-mgr, or MGS)
+- Ceph Metadata Service (ceph-mds, or MDS)
+- Ceph Object Storage Daemon (ceph-osd, or OSD)
TIP: We highly recommend to get familiar with Ceph
footnote:[Ceph intro {cephdocs-url}/start/intro/],
footnote:[Ceph glossary {cephdocs-url}/glossary].
-Precondition
-------------
+Recommendations for a Healthy Ceph Cluster
+------------------------------------------
-To build a hyper-converged Proxmox + Ceph Cluster, you must use at least
-three (preferably) identical servers for the setup.
+To build a hyper-converged Proxmox + Ceph Cluster, you must use at least three
+(preferably) identical servers for the setup.
Check also the recommendations from
{cephdocs-url}/start/hardware-recommendations/[Ceph's website].
+NOTE: The recommendations below should be seen as a rough guidance for choosing
+hardware. Therefore, it is still essential to adapt it to your specific needs.
+You should test your setup and monitor health and performance continuously.
+
.CPU
-A high CPU core frequency reduces latency and should be preferred. As a simple
-rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
-provide enough resources for stable and durable Ceph performance.
+Ceph services can be classified into two categories:
+* Intensive CPU usage, benefiting from high CPU base frequencies and multiple
+ cores. Members of that category are:
+** Object Storage Daemon (OSD) services
+** Meta Data Service (MDS) used for CephFS
+* Moderate CPU usage, not needing multiple CPU cores. These are:
+** Monitor (MON) services
+** Manager (MGR) services
+
+As a simple rule of thumb, you should assign at least one CPU core (or thread)
+to each Ceph service to provide the minimum resources required for stable and
+durable Ceph performance.
+
+For example, if you plan to run a Ceph monitor, a Ceph manager and 6 Ceph OSDs
+services on a node you should reserve 8 CPU cores purely for Ceph when targeting
+basic and stable performance.
+
+Note that OSDs CPU usage depend mostly from the disks performance. The higher
+the possible IOPS (**IO** **O**perations per **S**econd) of a disk, the more CPU
+can be utilized by a OSD service.
+For modern enterprise SSD disks, like NVMe's that can permanently sustain a high
+IOPS load over 100'000 with sub millisecond latency, each OSD can use multiple
+CPU threads, e.g., four to six CPU threads utilized per NVMe backed OSD is
+likely for very high performance disks.
.Memory
Especially in a hyper-converged setup, the memory consumption needs to be
-carefully monitored. In addition to the predicted memory usage of virtual
-machines and containers, you must also account for having enough memory
-available for Ceph to provide excellent and stable performance.
+carefully planned out and monitored. In addition to the predicted memory usage
+of virtual machines and containers, you must also account for having enough
+memory available for Ceph to provide excellent and stable performance.
As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
-by an OSD. Especially during recovery, re-balancing or backfilling.
+by an OSD. While the usage might be less under normal conditions, it will use
+most during critical operations like recovery, re-balancing or backfilling.
+That means that you should avoid maxing out your available memory already on
+normal operation, but rather leave some headroom to cope with outages.
-The daemon itself will use additional memory. The Bluestore backend of the
-daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the
-legacy Filestore backend uses the OS page cache and the memory consumption is
-generally related to PGs of an OSD daemon.
+The OSD service itself will use additional memory. The Ceph BlueStore backend of
+the daemon requires by default **3-5 GiB of memory** (adjustable).
.Network
-We recommend a network bandwidth of at least 10 GbE or more, which is used
-exclusively for Ceph. A meshed network setup
+We recommend a network bandwidth of at least 10 Gbps, or more, to be used
+exclusively for Ceph traffic. A meshed network setup
footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
-is also an option if there are no 10 GbE switches available.
-
-The volume of traffic, especially during recovery, will interfere with other
-services on the same network and may even break the {pve} cluster stack.
-
-Furthermore, you should estimate your bandwidth needs. While one HDD might not
-saturate a 1 Gb link, multiple HDD OSDs per node can, and modern NVMe SSDs will
-even saturate 10 Gbps of bandwidth quickly. Deploying a network capable of even
-more bandwidth will ensure that this isn't your bottleneck and won't be anytime
-soon. 25, 40 or even 100 Gbps are possible.
+is also an option for three to five node clusters, if there are no 10+ Gbps
+switches available.
+
+[IMPORTANT]
+The volume of traffic, especially during recovery, will interfere
+with other services on the same network, especially the latency sensitive {pve}
+corosync cluster stack can be affected, resulting in possible loss of cluster
+quorum. Moving the Ceph traffic to dedicated and physical separated networks
+will avoid such interference, not only for corosync, but also for the networking
+services provided by any virtual guests.
+
+For estimating your bandwidth needs, you need to take the performance of your
+disks into account.. While a single HDD might not saturate a 1 Gb link, multiple
+HDD OSDs per node can already saturate 10 Gbps too.
+If modern NVMe-attached SSDs are used, a single one can already saturate 10 Gbps
+of bandwidth, or more. For such high-performance setups we recommend at least
+a 25 Gpbs, while even 40 Gbps or 100+ Gbps might be required to utilize the full
+performance potential of the underlying disks.
+
+If unsure, we recommend using three (physical) separate networks for
+high-performance setups:
+* one very high bandwidth (25+ Gbps) network for Ceph (internal) cluster
+ traffic.
+* one high bandwidth (10+ Gpbs) network for Ceph (public) traffic between the
+ ceph server and ceph client storage traffic. Depending on your needs this can
+ also be used to host the virtual guest traffic and the VM live-migration
+ traffic.
+* one medium bandwidth (1 Gbps) exclusive for the latency sensitive corosync
+ cluster communication.
.Disks
When planning the size of your Ceph cluster, it is important to take the
and WAL / DB (or journal) disk must be selected, otherwise the faster disk
becomes the bottleneck for all linked OSDs.
-Aside from the disk type, Ceph performs best with an even sized and distributed
-amount of disks per node. For example, 4 x 500 GB disks within each node is
-better than a mixed setup with a single 1 TB and three 250 GB disk.
+Aside from the disk type, Ceph performs best with an evenly sized, and an evenly
+distributed amount of disks per node. For example, 4 x 500 GB disks within each
+node is better than a mixed setup with a single 1 TB and three 250 GB disk.
You also need to balance OSD count and single OSD capacity. More capacity
allows you to increase storage density, but it also means that a single OSD
WARNING: Avoid RAID controllers. Use host bus adapter (HBA) instead.
-NOTE: The above recommendations should be seen as a rough guidance for choosing
-hardware. Therefore, it is still essential to adapt it to your specific needs.
-You should test your setup and monitor health and performance continuously.
-
[[pve_ceph_install_wizard]]
Initial Ceph Installation & Configuration
-----------------------------------------
The configuration step includes the following settings:
-* *Public Network:* You can set up a dedicated network for Ceph. This
-setting is required. Separating your Ceph traffic is highly recommended.
-Otherwise, it could cause trouble with other latency dependent services,
-for example, cluster communication may decrease Ceph's performance.
+[[pve_ceph_wizard_networks]]
+
+* *Public Network:* This network will be used for public storage communication
+ (e.g., for virtual machines using a Ceph RBD backed disk, or a CephFS mount),
+ and communication between the different Ceph services. This setting is
+ required.
+ +
+ Separating your Ceph traffic from the {pve} cluster communication (corosync),
+ and possible the front-facing (public) networks of your virtual guests, is
+ highly recommended. Otherwise, Ceph's high-bandwidth IO-traffic could cause
+ interference with other low-latency dependent services.
[thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
-* *Cluster Network:* As an optional step, you can go even further and
-separate the xref:pve_ceph_osds[OSD] replication & heartbeat traffic
-as well. This will relieve the public network and could lead to
-significant performance improvements, especially in large clusters.
+* *Cluster Network:* Specify to separate the xref:pve_ceph_osds[OSD] replication
+ and heartbeat traffic as well. This setting is optional.
+ +
+ Using a physically separated network is recommended, as it will relieve the
+ Ceph public and the virtual guests network, while also providing a significant
+ Ceph performance improvements.
+ +
+ The Ceph cluster network can be configured and moved to another physically
+ separated network at a later time.
-You have two more options which are considered advanced and therefore
-should only changed if you know what you are doing.
+You have two more options which are considered advanced and therefore should
+only changed if you know what you are doing.
-* *Number of replicas*: Defines how often an object is replicated
-* *Minimum replicas*: Defines the minimum number of required replicas
-for I/O to be marked as complete.
+* *Number of replicas*: Defines how often an object is replicated.
+* *Minimum replicas*: Defines the minimum number of required replicas for I/O to
+ be marked as complete.
Additionally, you need to choose your first monitor node. This step is required.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Alternatively to the the recommended {pve} Ceph installation wizard available
-in the web-interface, you can use the following CLI command on each node:
+in the web interface, you can use the following CLI command on each node:
[source,bash]
----
Create OSDs
~~~~~~~~~~~
-You can create an OSD either via the {pve} web-interface or via the CLI using
+You can create an OSD either via the {pve} web interface or via the CLI using
`pveceph`. For example:
[source,bash]
Create and Edit Pools
~~~~~~~~~~~~~~~~~~~~~
-You can create and edit pools from the command line or the web-interface of any
+You can create and edit pools from the command line or the web interface of any
{pve} host under **Ceph -> Pools**.
When no options are given, we set a default of **128 PGs**, a **size of 3
.Example for creating a pool over the CLI
[source,bash]
----
-pveceph pool create <name> --add_storages
+pveceph pool create <pool-name> --add_storages
----
TIP: If you would also like to automatically define a storage for your
-pool, keep the `Add as Storage' checkbox checked in the web-interface, or use the
-command line option '--add_storages' at pool creation.
+pool, keep the `Add as Storage' checkbox checked in the web interface, or use the
+command-line option '--add_storages' at pool creation.
Pool Options
^^^^^^^^^^^^
[[pve_ceph_ec_pools]]
-Erasure Coded (EC) Pools
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-Erasure coded (EC) pools can offer more usable space for the price of
-performance. In replicated pools, multiple replicas of the data are stored
-('size'). In erasure coded pool, data is split into 'k' data chunks with
-additional 'm' coding chunks. The coding chunks can be used to recreate data
-should data chunks be missing. The number of coding chunks, 'm', defines how
-many OSDs can be lost without losing any data. The total amount of objects
-stored is 'k + m'.
-
-The default 'min_size' of an EC pool depends on the 'm' parameter. If 'm = 1',
-the 'min_size' of the EC pool will be 'k'. The 'min_size' will be 'k + 1' if
-'m > 1'. The Ceph documentation recommends a conservative 'min_size' of 'k + 2'
+Erasure Coded Pools
+~~~~~~~~~~~~~~~~~~~
+
+Erasure coding (EC) is a form of `forward error correction' codes that allows
+to recover from a certain amount of data loss. Erasure coded pools can offer
+more usable space compared to replicated pools, but they do that for the price
+of performance.
+
+For comparison: in classic, replicated pools, multiple replicas of the data
+are stored (`size`) while in erasure coded pool, data is split into `k` data
+chunks with additional `m` coding (checking) chunks. Those coding chunks can be
+used to recreate data should data chunks be missing.
+
+The number of coding chunks, `m`, defines how many OSDs can be lost without
+losing any data. The total amount of objects stored is `k + m`.
+
+Creating EC Pools
+^^^^^^^^^^^^^^^^^
+
+Erasure coded (EC) pools can be created with the `pveceph` CLI tooling.
+Planning an EC pool needs to account for the fact, that they work differently
+than replicated pools.
+
+The default `min_size` of an EC pool depends on the `m` parameter. If `m = 1`,
+the `min_size` of the EC pool will be `k`. The `min_size` will be `k + 1` if
+`m > 1`. The Ceph documentation recommends a conservative `min_size` of `k + 2`
footnote:[Ceph Erasure Coded Pool Recovery
{cephdocs-url}/rados/operations/erasure-code/#erasure-coded-pool-recovery].
-If there are less than 'min_size' OSDs available, any IO to the pool will be
+If there are less than `min_size` OSDs available, any IO to the pool will be
blocked until there are enough OSDs available again.
-NOTE: When planning an erasure coded pool, keep an eye on the 'min_size' as it
+NOTE: When planning an erasure coded pool, keep an eye on the `min_size` as it
defines how many OSDs need to be available. Otherwise, IO will be blocked.
-For example, an EC pool with 'k = 2' and 'm = 1' will have 'size = 3',
-'min_size = 2' and will stay operational if one OSD fails. If the pool is
-configured with 'k = 2', 'm = 2', it will have a 'size = 4' and 'min_size = 3'
+For example, an EC pool with `k = 2` and `m = 1` will have `size = 3`,
+`min_size = 2` and will stay operational if one OSD fails. If the pool is
+configured with `k = 2`, `m = 2`, it will have a `size = 4` and `min_size = 3`
and stay operational if one OSD is lost.
To create a new EC pool, run the following command:
[source,bash]
----
-pceveph pool create <pool name> --erasure-coding k=2,m=1
+pveceph pool create <pool-name> --erasure-coding k=2,m=1
----
-Optional parameters are 'failure-domain' and 'device-class'. If you
+Optional parameters are `failure-domain` and `device-class`. If you
need to change any EC profile settings used by the pool, you will have to
create a new pool with a new profile.
This will create a new EC pool plus the needed replicated pool to store the RBD
-omap and other metadata. In the end, there will be a '<pool name>-data' and
-'<pool name>-metada' pool. The default behavior is to create a matching storage
+omap and other metadata. In the end, there will be a `<pool name>-data` and
+`<pool name>-metada` pool. The default behavior is to create a matching storage
configuration as well. If that behavior is not wanted, you can disable it by
-providing the '--add_storages 0' parameter. When configuring the storage
-configuration manually, keep in mind that the 'data-pool' parameter needs to be
+providing the `--add_storages 0` parameter. When configuring the storage
+configuration manually, keep in mind that the `data-pool` parameter needs to be
set. Only then will the EC pool be used to store the data objects. For example:
-NOTE: The optional parameters '--size', '--min_size' and '--crush_rule' will be
+NOTE: The optional parameters `--size`, `--min_size` and `--crush_rule` will be
used for the replicated metadata pool, but not for the erasure coded data pool.
-If you need to change the 'min_size' on the data pool, you can do it later.
-The 'size' and 'crush_rule' parameters cannot be changed on erasure coded
+If you need to change the `min_size` on the data pool, you can do it later.
+The `size` and `crush_rule` parameters cannot be changed on erasure coded
pools.
-[source,bash]
-----
-pvesm add rbd <storage name> --pool <replicated pool> --data-pool <ec pool>
-----
-
If there is a need to further customize the EC profile, you can do so by
creating it with the Ceph tools directly footnote:[Ceph Erasure Code Profile
{cephdocs-url}/rados/operations/erasure-code/#erasure-code-profiles], and
-specify the profile to use with the 'profile' parameter.
+specify the profile to use with the `profile` parameter.
For example:
[source,bash]
----
-pceveph pool create <pool name> --erasure-coding profile=<profile name>
+pveceph pool create <pool-name> --erasure-coding profile=<profile-name>
+----
+
+Adding EC Pools as Storage
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can add an already existing EC pool as storage to {pve}. It works the same
+way as adding an `RBD` pool but requires the extra `data-pool` option.
+
+[source,bash]
+----
+pvesm add rbd <storage-name> --pool <replicated-pool> --data-pool <ec-pool>
----
+TIP: Do not forget to add the `keyring` and `monhost` option for any external
+Ceph clusters, not managed by the local {pve} cluster.
Destroy Pools
~~~~~~~~~~~~~
[frame="none",grid="none", align="left", cols="30%,70%"]
|===
|<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
-|<root>|which crush root it should belong to (default ceph root "default")
+|<root>|which crush root it should belong to (default Ceph root "default")
|<failure-domain>|at which failure-domain the objects should be distributed (usually host)
|<class>|what type of OSD backing store to use (e.g., nvme, ssd, hdd)
|===
- xref:pveceph_fs_mds[Setup at least one MDS]
After this is complete, you can simply create a CephFS through
-either the Web GUI's `Node -> CephFS` panel or the command line tool `pveceph`,
+either the Web GUI's `Node -> CephFS` panel or the command-line tool `pveceph`,
for example:
----
* Now make sure that no metadata server (`MDS`) is running for that CephFS,
either by stopping or destroying them. This can be done through the web
- interface or via the command line interface, for the latter you would issue
+ interface or via the command-line interface, for the latter you would issue
the following command:
+
----
pveceph fs destroy NAME --remove-storages --remove-pools
----
+
-This will automatically destroy the underlying ceph pools as well as remove
+This will automatically destroy the underlying Ceph pools as well as remove
the storages from pve config.
After these steps, the CephFS should be completely removed and if you have
projects / pve-docs.git / blobdiff