Deploy Hyper-Converged Ceph Cluster
===================================
:pve-toplevel:
+
+Introduction
+------------
endif::manvolnum[]
-[thumbnail="screenshot/gui-ceph-status.png"]
+[thumbnail="screenshot/gui-ceph-status-dashboard.png"]
{pve} unifies your compute and storage systems, that is, you can use the same
physical nodes within a cluster for both computing (processing VMs and
- Snapshot support
- Self healing
- Scalable to the exabyte level
+- Provides block, file system, and object storage
- Setup pools with different performance and redundancy characteristics
- Data is replicated, making it fault tolerant
- Runs on commodity hardware
- No need for hardware RAID controllers
- Open source
-For small to medium-sized deployments, it is possible to install a Ceph server for
-RADOS Block Devices (RBD) directly on your {pve} cluster nodes (see
-xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]). Recent
-hardware has a lot of CPU power and RAM, so running storage services
-and VMs on the same node is possible.
+For small to medium-sized deployments, it is possible to install a Ceph server
+for using RADOS Block Devices (RBD) or CephFS directly on your {pve} cluster
+nodes (see xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
+Recent hardware has a lot of CPU power and RAM, so running storage services and
+virtual guests on the same node is possible.
-To simplify management, we provide 'pveceph' - a tool for installing and
-managing {ceph} services on {pve} nodes.
+To simplify management, {pve} provides you native integration to install and
+manage {ceph} services on {pve} nodes either via the built-in web interface, or
+using the 'pveceph' command line tool.
+
+Terminology
+-----------
+
+// TODO: extend and also describe basic architecture here.
.Ceph consists of multiple Daemons, for use as an RBD storage:
-- Ceph Monitor (ceph-mon)
-- Ceph Manager (ceph-mgr)
-- Ceph OSD (ceph-osd; Object Storage Daemon)
+- Ceph Monitor (ceph-mon, or MON)
+- Ceph Manager (ceph-mgr, or MGS)
+- Ceph Metadata Service (ceph-mds, or MDS)
+- Ceph Object Storage Daemon (ceph-osd, or OSD)
TIP: We highly recommend to get familiar with Ceph
footnote:[Ceph intro {cephdocs-url}/start/intro/],
footnote:[Ceph glossary {cephdocs-url}/glossary].
-Precondition
-------------
+Recommendations for a Healthy Ceph Cluster
+------------------------------------------
-To build a hyper-converged Proxmox + Ceph Cluster, you must use at least
-three (preferably) identical servers for the setup.
+To build a hyper-converged Proxmox + Ceph Cluster, you must use at least three
+(preferably) identical servers for the setup.
Check also the recommendations from
{cephdocs-url}/start/hardware-recommendations/[Ceph's website].
+NOTE: The recommendations below should be seen as a rough guidance for choosing
+hardware. Therefore, it is still essential to adapt it to your specific needs.
+You should test your setup and monitor health and performance continuously.
+
.CPU
-A high CPU core frequency reduces latency and should be preferred. As a simple
-rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
-provide enough resources for stable and durable Ceph performance.
+Ceph services can be classified into two categories:
+* Intensive CPU usage, benefiting from high CPU base frequencies and multiple
+ cores. Members of that category are:
+** Object Storage Daemon (OSD) services
+** Meta Data Service (MDS) used for CephFS
+* Moderate CPU usage, not needing multiple CPU cores. These are:
+** Monitor (MON) services
+** Manager (MGR) services
+
+As a simple rule of thumb, you should assign at least one CPU core (or thread)
+to each Ceph service to provide the minimum resources required for stable and
+durable Ceph performance.
+
+For example, if you plan to run a Ceph monitor, a Ceph manager and 6 Ceph OSDs
+services on a node you should reserve 8 CPU cores purely for Ceph when targeting
+basic and stable performance.
+
+Note that OSDs CPU usage depend mostly from the disks performance. The higher
+the possible IOPS (**IO** **O**perations per **S**econd) of a disk, the more CPU
+can be utilized by a OSD service.
+For modern enterprise SSD disks, like NVMe's that can permanently sustain a high
+IOPS load over 100'000 with sub millisecond latency, each OSD can use multiple
+CPU threads, e.g., four to six CPU threads utilized per NVMe backed OSD is
+likely for very high performance disks.
.Memory
Especially in a hyper-converged setup, the memory consumption needs to be
-carefully monitored. In addition to the predicted memory usage of virtual
-machines and containers, you must also account for having enough memory
-available for Ceph to provide excellent and stable performance.
+carefully planned out and monitored. In addition to the predicted memory usage
+of virtual machines and containers, you must also account for having enough
+memory available for Ceph to provide excellent and stable performance.
As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
-by an OSD. Especially during recovery, rebalancing or backfilling.
+by an OSD. While the usage might be less under normal conditions, it will use
+most during critical operations like recovery, re-balancing or backfilling.
+That means that you should avoid maxing out your available memory already on
+normal operation, but rather leave some headroom to cope with outages.
-The daemon itself will use additional memory. The Bluestore backend of the
-daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the
-legacy Filestore backend uses the OS page cache and the memory consumption is
-generally related to PGs of an OSD daemon.
+The OSD service itself will use additional memory. The Ceph BlueStore backend of
+the daemon requires by default **3-5 GiB of memory** (adjustable).
.Network
-We recommend a network bandwidth of at least 10 GbE or more, which is used
-exclusively for Ceph. A meshed network setup
+We recommend a network bandwidth of at least 10 Gbps, or more, to be used
+exclusively for Ceph traffic. A meshed network setup
footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
-is also an option if there are no 10 GbE switches available.
-
-The volume of traffic, especially during recovery, will interfere with other
-services on the same network and may even break the {pve} cluster stack.
-
-Furthermore, you should estimate your bandwidth needs. While one HDD might not
-saturate a 1 Gb link, multiple HDD OSDs per node can, and modern NVMe SSDs will
-even saturate 10 Gbps of bandwidth quickly. Deploying a network capable of even
-more bandwidth will ensure that this isn't your bottleneck and won't be anytime
-soon. 25, 40 or even 100 Gbps are possible.
+is also an option for three to five node clusters, if there are no 10+ Gbps
+switches available.
+
+[IMPORTANT]
+The volume of traffic, especially during recovery, will interfere
+with other services on the same network, especially the latency sensitive {pve}
+corosync cluster stack can be affected, resulting in possible loss of cluster
+quorum. Moving the Ceph traffic to dedicated and physical separated networks
+will avoid such interference, not only for corosync, but also for the networking
+services provided by any virtual guests.
+
+For estimating your bandwidth needs, you need to take the performance of your
+disks into account.. While a single HDD might not saturate a 1 Gb link, multiple
+HDD OSDs per node can already saturate 10 Gbps too.
+If modern NVMe-attached SSDs are used, a single one can already saturate 10 Gbps
+of bandwidth, or more. For such high-performance setups we recommend at least
+a 25 Gpbs, while even 40 Gbps or 100+ Gbps might be required to utilize the full
+performance potential of the underlying disks.
+
+If unsure, we recommend using three (physical) separate networks for
+high-performance setups:
+* one very high bandwidth (25+ Gbps) network for Ceph (internal) cluster
+ traffic.
+* one high bandwidth (10+ Gpbs) network for Ceph (public) traffic between the
+ ceph server and ceph client storage traffic. Depending on your needs this can
+ also be used to host the virtual guest traffic and the VM live-migration
+ traffic.
+* one medium bandwidth (1 Gbps) exclusive for the latency sensitive corosync
+ cluster communication.
.Disks
When planning the size of your Ceph cluster, it is important to take the
setups to reduce recovery time, minimizing the likelihood of a subsequent
failure event during recovery.
-In general SSDs will provide more IOPs than spinning disks. With this in mind,
+In general, SSDs will provide more IOPS than spinning disks. With this in mind,
in addition to the higher cost, it may make sense to implement a
xref:pve_ceph_device_classes[class based] separation of pools. Another way to
speed up OSDs is to use a faster disk as a journal or
and WAL / DB (or journal) disk must be selected, otherwise the faster disk
becomes the bottleneck for all linked OSDs.
-Aside from the disk type, Ceph performs best with an even sized and distributed
-amount of disks per node. For example, 4 x 500 GB disks within each node is
-better than a mixed setup with a single 1 TB and three 250 GB disk.
+Aside from the disk type, Ceph performs best with an evenly sized, and an evenly
+distributed amount of disks per node. For example, 4 x 500 GB disks within each
+node is better than a mixed setup with a single 1 TB and three 250 GB disk.
You also need to balance OSD count and single OSD capacity. More capacity
allows you to increase storage density, but it also means that a single OSD
WARNING: Avoid RAID controllers. Use host bus adapter (HBA) instead.
-NOTE: The above recommendations should be seen as a rough guidance for choosing
-hardware. Therefore, it is still essential to adapt it to your specific needs.
-You should test your setup and monitor health and performance continuously.
-
[[pve_ceph_install_wizard]]
Initial Ceph Installation & Configuration
-----------------------------------------
After starting the installation, the wizard will download and install all the
required packages from {pve}'s Ceph repository.
+[thumbnail="screenshot/gui-node-ceph-install-wizard-step0.png"]
After finishing the installation step, you will need to create a configuration.
This step is only needed once per cluster, as this configuration is distributed
The configuration step includes the following settings:
-* *Public Network:* You can set up a dedicated network for Ceph. This
-setting is required. Separating your Ceph traffic is highly recommended.
-Otherwise, it could cause trouble with other latency dependent services,
-for example, cluster communication may decrease Ceph's performance.
+[[pve_ceph_wizard_networks]]
+
+* *Public Network:* This network will be used for public storage communication
+ (e.g., for virtual machines using a Ceph RBD backed disk, or a CephFS mount),
+ and communication between the different Ceph services. This setting is
+ required.
+ +
+ Separating your Ceph traffic from the {pve} cluster communication (corosync),
+ and possible the front-facing (public) networks of your virtual guests, is
+ highly recommended. Otherwise, Ceph's high-bandwidth IO-traffic could cause
+ interference with other low-latency dependent services.
[thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
-* *Cluster Network:* As an optional step, you can go even further and
-separate the xref:pve_ceph_osds[OSD] replication & heartbeat traffic
-as well. This will relieve the public network and could lead to
-significant performance improvements, especially in large clusters.
+* *Cluster Network:* Specify to separate the xref:pve_ceph_osds[OSD] replication
+ and heartbeat traffic as well. This setting is optional.
+ +
+ Using a physically separated network is recommended, as it will relieve the
+ Ceph public and the virtual guests network, while also providing a significant
+ Ceph performance improvements.
+ +
+ The Ceph cluster network can be configured and moved to another physically
+ separated network at a later time.
-You have two more options which are considered advanced and therefore
-should only changed if you know what you are doing.
+You have two more options which are considered advanced and therefore should
+only changed if you know what you are doing.
-* *Number of replicas*: Defines how often an object is replicated
-* *Minimum replicas*: Defines the minimum number of required replicas
-for I/O to be marked as complete.
+* *Number of replicas*: Defines how often an object is replicated.
+* *Minimum replicas*: Defines the minimum number of required replicas for I/O to
+ be marked as complete.
Additionally, you need to choose your first monitor node. This step is required.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Alternatively to the the recommended {pve} Ceph installation wizard available
-in the web-interface, you can use the following CLI command on each node:
+in the web interface, you can use the following CLI command on each node:
[source,bash]
----
Create OSDs
~~~~~~~~~~~
-You can create an OSD either via the {pve} web-interface or via the CLI using
+You can create an OSD either via the {pve} web interface or via the CLI using
`pveceph`. For example:
[source,bash]
[[pve_ceph_pools]]
Ceph Pools
----------
+
+[thumbnail="screenshot/gui-ceph-pools.png"]
+
A pool is a logical group for storing objects. It holds a collection of objects,
known as **P**lacement **G**roups (`PG`, `pg_num`).
Create and Edit Pools
~~~~~~~~~~~~~~~~~~~~~
-You can create and edit pools from the command line or the web-interface of any
+You can create and edit pools from the command line or the web interface of any
{pve} host under **Ceph -> Pools**.
-[thumbnail="screenshot/gui-ceph-pools.png"]
-
When no options are given, we set a default of **128 PGs**, a **size of 3
replicas** and a **min_size of 2 replicas**, to ensure no data loss occurs if
any OSD fails.
It is advised that you either enable the PG-Autoscaler or calculate the PG
number based on your setup. You can find the formula and the PG calculator
-footnote:[PG calculator https://ceph.com/pgcalc/] online. From Ceph Nautilus
+footnote:[PG calculator https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/] online. From Ceph Nautilus
onward, you can change the number of PGs
footnoteref:[placement_groups,Placement Groups
{cephdocs-url}/rados/operations/placement-groups/] after the setup.
.Example for creating a pool over the CLI
[source,bash]
----
-pveceph pool create <name> --add_storages
+pveceph pool create <pool-name> --add_storages
----
TIP: If you would also like to automatically define a storage for your
-pool, keep the `Add as Storage' checkbox checked in the web-interface, or use the
-command line option '--add_storages' at pool creation.
+pool, keep the `Add as Storage' checkbox checked in the web interface, or use the
+command-line option '--add_storages' at pool creation.
Pool Options
^^^^^^^^^^^^
+[thumbnail="screenshot/gui-ceph-pool-create.png"]
+
The following options are available on pool creation, and partially also when
editing a pool.
manual.
+[[pve_ceph_ec_pools]]
+Erasure Coded Pools
+~~~~~~~~~~~~~~~~~~~
+
+Erasure coding (EC) is a form of `forward error correction' codes that allows
+to recover from a certain amount of data loss. Erasure coded pools can offer
+more usable space compared to replicated pools, but they do that for the price
+of performance.
+
+For comparison: in classic, replicated pools, multiple replicas of the data
+are stored (`size`) while in erasure coded pool, data is split into `k` data
+chunks with additional `m` coding (checking) chunks. Those coding chunks can be
+used to recreate data should data chunks be missing.
+
+The number of coding chunks, `m`, defines how many OSDs can be lost without
+losing any data. The total amount of objects stored is `k + m`.
+
+Creating EC Pools
+^^^^^^^^^^^^^^^^^
+
+Erasure coded (EC) pools can be created with the `pveceph` CLI tooling.
+Planning an EC pool needs to account for the fact, that they work differently
+than replicated pools.
+
+The default `min_size` of an EC pool depends on the `m` parameter. If `m = 1`,
+the `min_size` of the EC pool will be `k`. The `min_size` will be `k + 1` if
+`m > 1`. The Ceph documentation recommends a conservative `min_size` of `k + 2`
+footnote:[Ceph Erasure Coded Pool Recovery
+{cephdocs-url}/rados/operations/erasure-code/#erasure-coded-pool-recovery].
+
+If there are less than `min_size` OSDs available, any IO to the pool will be
+blocked until there are enough OSDs available again.
+
+NOTE: When planning an erasure coded pool, keep an eye on the `min_size` as it
+defines how many OSDs need to be available. Otherwise, IO will be blocked.
+
+For example, an EC pool with `k = 2` and `m = 1` will have `size = 3`,
+`min_size = 2` and will stay operational if one OSD fails. If the pool is
+configured with `k = 2`, `m = 2`, it will have a `size = 4` and `min_size = 3`
+and stay operational if one OSD is lost.
+
+To create a new EC pool, run the following command:
+
+[source,bash]
+----
+pveceph pool create <pool-name> --erasure-coding k=2,m=1
+----
+
+Optional parameters are `failure-domain` and `device-class`. If you
+need to change any EC profile settings used by the pool, you will have to
+create a new pool with a new profile.
+
+This will create a new EC pool plus the needed replicated pool to store the RBD
+omap and other metadata. In the end, there will be a `<pool name>-data` and
+`<pool name>-metada` pool. The default behavior is to create a matching storage
+configuration as well. If that behavior is not wanted, you can disable it by
+providing the `--add_storages 0` parameter. When configuring the storage
+configuration manually, keep in mind that the `data-pool` parameter needs to be
+set. Only then will the EC pool be used to store the data objects. For example:
+
+NOTE: The optional parameters `--size`, `--min_size` and `--crush_rule` will be
+used for the replicated metadata pool, but not for the erasure coded data pool.
+If you need to change the `min_size` on the data pool, you can do it later.
+The `size` and `crush_rule` parameters cannot be changed on erasure coded
+pools.
+
+If there is a need to further customize the EC profile, you can do so by
+creating it with the Ceph tools directly footnote:[Ceph Erasure Code Profile
+{cephdocs-url}/rados/operations/erasure-code/#erasure-code-profiles], and
+specify the profile to use with the `profile` parameter.
+
+For example:
+[source,bash]
+----
+pveceph pool create <pool-name> --erasure-coding profile=<profile-name>
+----
+
+Adding EC Pools as Storage
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can add an already existing EC pool as storage to {pve}. It works the same
+way as adding an `RBD` pool but requires the extra `data-pool` option.
+
+[source,bash]
+----
+pvesm add rbd <storage-name> --pool <replicated-pool> --data-pool <ec-pool>
+----
+
+TIP: Do not forget to add the `keyring` and `monhost` option for any external
+Ceph clusters, not managed by the local {pve} cluster.
+
Destroy Pools
~~~~~~~~~~~~~
section CRUSH map footnote:[CRUSH map {cephdocs-url}/rados/operations/crush-map/].
This map can be altered to reflect different replication hierarchies. The object
-replicas can be separated (eg. failure domains), while maintaining the desired
+replicas can be separated (e.g., failure domains), while maintaining the desired
distribution.
A common configuration is to use different classes of disks for different Ceph
[frame="none",grid="none", align="left", cols="30%,70%"]
|===
|<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
-|<root>|which crush root it should belong to (default ceph root "default")
+|<root>|which crush root it should belong to (default Ceph root "default")
|<failure-domain>|at which failure-domain the objects should be distributed (usually host)
-|<class>|what type of OSD backing store to use (eg. nvme, ssd, hdd)
+|<class>|what type of OSD backing store to use (e.g., nvme, ssd, hdd)
|===
Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
- xref:pveceph_fs_mds[Setup at least one MDS]
After this is complete, you can simply create a CephFS through
-either the Web GUI's `Node -> CephFS` panel or the command line tool `pveceph`,
+either the Web GUI's `Node -> CephFS` panel or the command-line tool `pveceph`,
for example:
----
WARNING: Destroying a CephFS will render all of its data unusable. This cannot be
undone!
-If you really want to destroy an existing CephFS, you first need to stop or
-destroy all metadata servers (`MĚ€DS`). You can destroy them either via the web
-interface or via the command line interface, by issuing
+To completely and gracefully remove a CephFS, the following steps are
+necessary:
+* Disconnect every non-{PVE} client (e.g. unmount the CephFS in guests).
+* Disable all related CephFS {PVE} storage entries (to prevent it from being
+ automatically mounted).
+* Remove all used resources from guests (e.g. ISOs) that are on the CephFS you
+ want to destroy.
+* Unmount the CephFS storages on all cluster nodes manually with
++
----
-pveceph mds destroy NAME
+umount /mnt/pve/<STORAGE-NAME>
----
-on each {pve} node hosting an MDS daemon.
-
-Then, you can remove (destroy) the CephFS by issuing
-
++
+Where `<STORAGE-NAME>` is the name of the CephFS storage in your {PVE}.
+
+* Now make sure that no metadata server (`MDS`) is running for that CephFS,
+ either by stopping or destroying them. This can be done through the web
+ interface or via the command-line interface, for the latter you would issue
+ the following command:
++
----
-ceph fs rm NAME --yes-i-really-mean-it
+pveceph stop --service mds.NAME
----
-on a single node hosting Ceph. After this, you may want to remove the created
-data and metadata pools, this can be done either over the Web GUI or the CLI
-with:
-
++
+to stop them, or
++
+----
+pveceph mds destroy NAME
+----
++
+to destroy them.
++
+Note that standby servers will automatically be promoted to active when an
+active `MDS` is stopped or removed, so it is best to first stop all standby
+servers.
+
+* Now you can destroy the CephFS with
++
----
-pveceph pool destroy NAME
+pveceph fs destroy NAME --remove-storages --remove-pools
----
++
+This will automatically destroy the underlying Ceph pools as well as remove
+the storages from pve config.
+After these steps, the CephFS should be completely removed and if you have
+other CephFS instances, the stopped metadata servers can be started again
+to act as standbys.
Ceph maintenance
----------------
projects / pve-docs.git / blobdiff