Deploy Hyper-Converged Ceph Cluster
===================================
:pve-toplevel:
+
+Introduction
+------------
endif::manvolnum[]
-[thumbnail="screenshot/gui-ceph-status.png"]
+[thumbnail="screenshot/gui-ceph-status-dashboard.png"]
-{pve} unifies your compute and storage systems, i.e. you can use the same
+{pve} unifies your compute and storage systems, that is, you can use the same
physical nodes within a cluster for both computing (processing VMs and
containers) and replicated storage. The traditional silos of compute and
storage resources can be wrapped up into a single hyper-converged appliance.
-Separate storage networks (SANs) and connections via network attached storages
+Separate storage networks (SANs) and connections via network attached storage
(NAS) disappear. With the integration of Ceph, an open source software-defined
storage platform, {pve} has the ability to run and manage Ceph storage directly
on the hypervisor nodes.
excellent performance, reliability and scalability.
.Some advantages of Ceph on {pve} are:
-- Easy setup and management with CLI and GUI support
+- Easy setup and management via CLI and GUI
- Thin provisioning
-- Snapshots support
+- Snapshot support
- Self healing
- Scalable to the exabyte level
+- Provides block, file system, and object storage
- Setup pools with different performance and redundancy characteristics
- Data is replicated, making it fault tolerant
-- Runs on economical commodity hardware
+- Runs on commodity hardware
- No need for hardware RAID controllers
- Open source
-For small to mid sized deployments, it is possible to install a Ceph server for
-RADOS Block Devices (RBD) directly on your {pve} cluster nodes, see
-xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
-hardware has plenty of CPU power and RAM, so running storage services
-and VMs on the same node is possible.
+For small to medium-sized deployments, it is possible to install a Ceph server
+for using RADOS Block Devices (RBD) or CephFS directly on your {pve} cluster
+nodes (see xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
+Recent hardware has a lot of CPU power and RAM, so running storage services and
+virtual guests on the same node is possible.
-To simplify management, we provide 'pveceph' - a tool to install and
-manage {ceph} services on {pve} nodes.
+To simplify management, {pve} provides you native integration to install and
+manage {ceph} services on {pve} nodes either via the built-in web interface, or
+using the 'pveceph' command line tool.
-.Ceph consists of a couple of Daemons footnote:[Ceph intro https://docs.ceph.com/docs/{ceph_codename}/start/intro/], for use as a RBD storage:
-- Ceph Monitor (ceph-mon)
-- Ceph Manager (ceph-mgr)
-- Ceph OSD (ceph-osd; Object Storage Daemon)
-TIP: We highly recommend to get familiar with Ceph's architecture
-footnote:[Ceph architecture https://docs.ceph.com/docs/{ceph_codename}/architecture/]
+Terminology
+-----------
+
+// TODO: extend and also describe basic architecture here.
+.Ceph consists of multiple Daemons, for use as an RBD storage:
+- Ceph Monitor (ceph-mon, or MON)
+- Ceph Manager (ceph-mgr, or MGS)
+- Ceph Metadata Service (ceph-mds, or MDS)
+- Ceph Object Storage Daemon (ceph-osd, or OSD)
+
+TIP: We highly recommend to get familiar with Ceph
+footnote:[Ceph intro {cephdocs-url}/start/intro/],
+its architecture
+footnote:[Ceph architecture {cephdocs-url}/architecture/]
and vocabulary
-footnote:[Ceph glossary https://docs.ceph.com/docs/{ceph_codename}/glossary].
+footnote:[Ceph glossary {cephdocs-url}/glossary].
-Precondition
-------------
+Recommendations for a Healthy Ceph Cluster
+------------------------------------------
-To build a hyper-converged Proxmox + Ceph Cluster there should be at least
-three (preferably) identical servers for the setup.
+To build a hyper-converged Proxmox + Ceph Cluster, you must use at least three
+(preferably) identical servers for the setup.
Check also the recommendations from
-https://docs.ceph.com/docs/{ceph_codename}/start/hardware-recommendations/[Ceph's website].
+{cephdocs-url}/start/hardware-recommendations/[Ceph's website].
+
+NOTE: The recommendations below should be seen as a rough guidance for choosing
+hardware. Therefore, it is still essential to adapt it to your specific needs.
+You should test your setup and monitor health and performance continuously.
.CPU
-Higher CPU core frequency reduce latency and should be preferred. As a simple
-rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
-provide enough resources for stable and durable Ceph performance.
+Ceph services can be classified into two categories:
+* Intensive CPU usage, benefiting from high CPU base frequencies and multiple
+ cores. Members of that category are:
+** Object Storage Daemon (OSD) services
+** Meta Data Service (MDS) used for CephFS
+* Moderate CPU usage, not needing multiple CPU cores. These are:
+** Monitor (MON) services
+** Manager (MGR) services
+
+As a simple rule of thumb, you should assign at least one CPU core (or thread)
+to each Ceph service to provide the minimum resources required for stable and
+durable Ceph performance.
+
+For example, if you plan to run a Ceph monitor, a Ceph manager and 6 Ceph OSDs
+services on a node you should reserve 8 CPU cores purely for Ceph when targeting
+basic and stable performance.
+
+Note that OSDs CPU usage depend mostly from the disks performance. The higher
+the possible IOPS (**IO** **O**perations per **S**econd) of a disk, the more CPU
+can be utilized by a OSD service.
+For modern enterprise SSD disks, like NVMe's that can permanently sustain a high
+IOPS load over 100'000 with sub millisecond latency, each OSD can use multiple
+CPU threads, e.g., four to six CPU threads utilized per NVMe backed OSD is
+likely for very high performance disks.
.Memory
Especially in a hyper-converged setup, the memory consumption needs to be
-carefully monitored. In addition to the intended workload from virtual machines
-and containers, Ceph needs enough memory available to provide excellent and
-stable performance.
+carefully planned out and monitored. In addition to the predicted memory usage
+of virtual machines and containers, you must also account for having enough
+memory available for Ceph to provide excellent and stable performance.
As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
-by an OSD. Especially during recovery, rebalancing or backfilling.
+by an OSD. While the usage might be less under normal conditions, it will use
+most during critical operations like recovery, re-balancing or backfilling.
+That means that you should avoid maxing out your available memory already on
+normal operation, but rather leave some headroom to cope with outages.
-The daemon itself will use additional memory. The Bluestore backend of the
-daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the
-legacy Filestore backend uses the OS page cache and the memory consumption is
-generally related to PGs of an OSD daemon.
+The OSD service itself will use additional memory. The Ceph BlueStore backend of
+the daemon requires by default **3-5 GiB of memory** (adjustable).
.Network
-We recommend a network bandwidth of at least 10 GbE or more, which is used
-exclusively for Ceph. A meshed network setup
+We recommend a network bandwidth of at least 10 Gbps, or more, to be used
+exclusively for Ceph traffic. A meshed network setup
footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
-is also an option if there are no 10 GbE switches available.
-
-The volume of traffic, especially during recovery, will interfere with other
-services on the same network and may even break the {pve} cluster stack.
-
-Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
-link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate
-10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwidth
-will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or
-even 100 GBps are possible.
+is also an option for three to five node clusters, if there are no 10+ Gbps
+switches available.
+
+[IMPORTANT]
+The volume of traffic, especially during recovery, will interfere
+with other services on the same network, especially the latency sensitive {pve}
+corosync cluster stack can be affected, resulting in possible loss of cluster
+quorum. Moving the Ceph traffic to dedicated and physical separated networks
+will avoid such interference, not only for corosync, but also for the networking
+services provided by any virtual guests.
+
+For estimating your bandwidth needs, you need to take the performance of your
+disks into account.. While a single HDD might not saturate a 1 Gb link, multiple
+HDD OSDs per node can already saturate 10 Gbps too.
+If modern NVMe-attached SSDs are used, a single one can already saturate 10 Gbps
+of bandwidth, or more. For such high-performance setups we recommend at least
+a 25 Gpbs, while even 40 Gbps or 100+ Gbps might be required to utilize the full
+performance potential of the underlying disks.
+
+If unsure, we recommend using three (physical) separate networks for
+high-performance setups:
+* one very high bandwidth (25+ Gbps) network for Ceph (internal) cluster
+ traffic.
+* one high bandwidth (10+ Gpbs) network for Ceph (public) traffic between the
+ ceph server and ceph client storage traffic. Depending on your needs this can
+ also be used to host the virtual guest traffic and the VM live-migration
+ traffic.
+* one medium bandwidth (1 Gbps) exclusive for the latency sensitive corosync
+ cluster communication.
.Disks
When planning the size of your Ceph cluster, it is important to take the
-recovery time into consideration. Especially with small clusters, the recovery
+recovery time into consideration. Especially with small clusters, recovery
might take long. It is recommended that you use SSDs instead of HDDs in small
setups to reduce recovery time, minimizing the likelihood of a subsequent
failure event during recovery.
-In general SSDs will provide more IOPs than spinning disks. This fact and the
-higher cost may make a xref:pve_ceph_device_classes[class based] separation of
-pools appealing. Another possibility to speedup OSDs is to use a faster disk
-as journal or DB/**W**rite-**A**head-**L**og device, see
-xref:pve_ceph_osds[creating Ceph OSDs]. If a faster disk is used for multiple
-OSDs, a proper balance between OSD and WAL / DB (or journal) disk must be
-selected, otherwise the faster disk becomes the bottleneck for all linked OSDs.
+In general, SSDs will provide more IOPS than spinning disks. With this in mind,
+in addition to the higher cost, it may make sense to implement a
+xref:pve_ceph_device_classes[class based] separation of pools. Another way to
+speed up OSDs is to use a faster disk as a journal or
+DB/**W**rite-**A**head-**L**og device, see
+xref:pve_ceph_osds[creating Ceph OSDs].
+If a faster disk is used for multiple OSDs, a proper balance between OSD
+and WAL / DB (or journal) disk must be selected, otherwise the faster disk
+becomes the bottleneck for all linked OSDs.
-Aside from the disk type, Ceph best performs with an even sized and distributed
-amount of disks per node. For example, 4 x 500 GB disks with in each node is
-better than a mixed setup with a single 1 TB and three 250 GB disk.
+Aside from the disk type, Ceph performs best with an evenly sized, and an evenly
+distributed amount of disks per node. For example, 4 x 500 GB disks within each
+node is better than a mixed setup with a single 1 TB and three 250 GB disk.
-One also need to balance OSD count and single OSD capacity. More capacity
-allows to increase storage density, but it also means that a single OSD
-failure forces ceph to recover more data at once.
+You also need to balance OSD count and single OSD capacity. More capacity
+allows you to increase storage density, but it also means that a single OSD
+failure forces Ceph to recover more data at once.
.Avoid RAID
As Ceph handles data object redundancy and multiple parallel writes to disks
(OSDs) on its own, using a RAID controller normally doesn’t improve
performance or availability. On the contrary, Ceph is designed to handle whole
-disks on it's own, without any abstraction in between. RAID controller are not
-designed for the Ceph use case and may complicate things and sometimes even
+disks on it's own, without any abstraction in between. RAID controllers are not
+designed for the Ceph workload and may complicate things and sometimes even
reduce performance, as their write and caching algorithms may interfere with
the ones from Ceph.
-WARNING: Avoid RAID controller, use host bus adapter (HBA) instead.
-
-NOTE: Above recommendations should be seen as a rough guidance for choosing
-hardware. Therefore, it is still essential to adapt it to your specific needs,
-test your setup and monitor health and performance continuously.
+WARNING: Avoid RAID controllers. Use host bus adapter (HBA) instead.
[[pve_ceph_install_wizard]]
-Initial Ceph installation & configuration
+Initial Ceph Installation & Configuration
-----------------------------------------
+Using the Web-based Wizard
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
[thumbnail="screenshot/gui-node-ceph-install.png"]
With {pve} you have the benefit of an easy to use installation wizard
for Ceph. Click on one of your cluster nodes and navigate to the Ceph
-section in the menu tree. If Ceph is not already installed you will be
-offered to do so now.
+section in the menu tree. If Ceph is not already installed, you will see a
+prompt offering to do so.
+
+The wizard is divided into multiple sections, where each needs to
+finish successfully, in order to use Ceph.
-The wizard is divided into different sections, where each needs to be
-finished successfully in order to use Ceph. After starting the installation
-the wizard will download and install all required packages from {pve}'s ceph
-repository.
+First you need to chose which Ceph version you want to install. Prefer the one
+from your other nodes, or the newest if this is the first node you install
+Ceph.
-After finishing the first step, you will need to create a configuration.
+After starting the installation, the wizard will download and install all the
+required packages from {pve}'s Ceph repository.
+[thumbnail="screenshot/gui-node-ceph-install-wizard-step0.png"]
+
+After finishing the installation step, you will need to create a configuration.
This step is only needed once per cluster, as this configuration is distributed
automatically to all remaining cluster members through {pve}'s clustered
xref:chapter_pmxcfs[configuration file system (pmxcfs)].
The configuration step includes the following settings:
-* *Public Network:* You should setup a dedicated network for Ceph, this
-setting is required. Separating your Ceph traffic is highly recommended,
-because it could lead to troubles with other latency dependent services,
-e.g., cluster communication may decrease Ceph's performance, if not done.
-
-[thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
+[[pve_ceph_wizard_networks]]
-* *Cluster Network:* As an optional step you can go even further and
-separate the xref:pve_ceph_osds[OSD] replication & heartbeat traffic
-as well. This will relieve the public network and could lead to
-significant performance improvements especially in big clusters.
+* *Public Network:* This network will be used for public storage communication
+ (e.g., for virtual machines using a Ceph RBD backed disk, or a CephFS mount),
+ and communication between the different Ceph services. This setting is
+ required.
+ +
+ Separating your Ceph traffic from the {pve} cluster communication (corosync),
+ and possible the front-facing (public) networks of your virtual guests, is
+ highly recommended. Otherwise, Ceph's high-bandwidth IO-traffic could cause
+ interference with other low-latency dependent services.
-You have two more options which are considered advanced and therefore
-should only changed if you are an expert.
-
-* *Number of replicas*: Defines the how often a object is replicated
-* *Minimum replicas*: Defines the minimum number of required replicas
- for I/O to be marked as complete.
-
-Additionally you need to choose your first monitor node, this is required.
-
-That's it, you should see a success page as the last step with further
-instructions on how to go on. You are now prepared to start using Ceph,
-even though you will need to create additional xref:pve_ceph_monitors[monitors],
-create some xref:pve_ceph_osds[OSDs] and at least one xref:pve_ceph_pools[pool].
+[thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
-The rest of this chapter will guide you on how to get the most out of
-your {pve} based Ceph setup, this will include aforementioned and
-more like xref:pveceph_fs[CephFS] which is a very handy addition to your
+* *Cluster Network:* Specify to separate the xref:pve_ceph_osds[OSD] replication
+ and heartbeat traffic as well. This setting is optional.
+ +
+ Using a physically separated network is recommended, as it will relieve the
+ Ceph public and the virtual guests network, while also providing a significant
+ Ceph performance improvements.
+ +
+ The Ceph cluster network can be configured and moved to another physically
+ separated network at a later time.
+
+You have two more options which are considered advanced and therefore should
+only changed if you know what you are doing.
+
+* *Number of replicas*: Defines how often an object is replicated.
+* *Minimum replicas*: Defines the minimum number of required replicas for I/O to
+ be marked as complete.
+
+Additionally, you need to choose your first monitor node. This step is required.
+
+That's it. You should now see a success page as the last step, with further
+instructions on how to proceed. Your system is now ready to start using Ceph.
+To get started, you will need to create some additional xref:pve_ceph_monitors[monitors],
+xref:pve_ceph_osds[OSDs] and at least one xref:pve_ceph_pools[pool].
+
+The rest of this chapter will guide you through getting the most out of
+your {pve} based Ceph setup. This includes the aforementioned tips and
+more, such as xref:pveceph_fs[CephFS], which is a helpful addition to your
new Ceph cluster.
[[pve_ceph_install]]
-Installation of Ceph Packages
------------------------------
-Use {pve} Ceph installation wizard (recommended) or run the following
-command on each node:
+CLI Installation of Ceph Packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alternatively to the the recommended {pve} Ceph installation wizard available
+in the web interface, you can use the following CLI command on each node:
[source,bash]
----
`/etc/apt/sources.list.d/ceph.list` and installs the required software.
-Create initial Ceph configuration
----------------------------------
-
-[thumbnail="screenshot/gui-ceph-config.png"]
+Initial Ceph configuration via CLI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the {pve} Ceph installation wizard (recommended) or run the
following command on one node:
----
This creates an initial configuration at `/etc/pve/ceph.conf` with a
-dedicated network for ceph. That file is automatically distributed to
-all {pve} nodes by using xref:chapter_pmxcfs[pmxcfs]. The command also
-creates a symbolic link from `/etc/ceph/ceph.conf` pointing to that file.
-So you can simply run Ceph commands without the need to specify a
+dedicated network for Ceph. This file is automatically distributed to
+all {pve} nodes, using xref:chapter_pmxcfs[pmxcfs]. The command also
+creates a symbolic link at `/etc/ceph/ceph.conf`, which points to that file.
+Thus, you can simply run Ceph commands without the need to specify a
configuration file.
[[pve_ceph_monitors]]
Ceph Monitor
-----------
-The Ceph Monitor (MON)
-footnote:[Ceph Monitor https://docs.ceph.com/docs/{ceph_codename}/start/intro/]
-maintains a master copy of the cluster map. For high availability you need to
-have at least 3 monitors. One monitor will already be installed if you
-used the installation wizard. You won't need more than 3 monitors as long
-as your cluster is small to midsize, only really large clusters will
-need more than that.
+[thumbnail="screenshot/gui-ceph-monitor.png"]
+
+The Ceph Monitor (MON)
+footnote:[Ceph Monitor {cephdocs-url}/start/intro/]
+maintains a master copy of the cluster map. For high availability, you need at
+least 3 monitors. One monitor will already be installed if you
+used the installation wizard. You won't need more than 3 monitors, as long
+as your cluster is small to medium-sized. Only really large clusters will
+require more than this.
[[pveceph_create_mon]]
Create Monitors
~~~~~~~~~~~~~~~
-[thumbnail="screenshot/gui-ceph-monitor.png"]
-
On each node where you want to place a monitor (three monitors are recommended),
-create it by using the 'Ceph -> Monitor' tab in the GUI or run.
+create one by using the 'Ceph -> Monitor' tab in the GUI or run:
[source,bash]
Destroy Monitors
~~~~~~~~~~~~~~~~
-To remove a Ceph Monitor via the GUI first select a node in the tree view and
+To remove a Ceph Monitor via the GUI, first select a node in the tree view and
go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**
button.
-To remove a Ceph Monitor via the CLI first connect to the node on which the MON
+To remove a Ceph Monitor via the CLI, first connect to the node on which the MON
is running. Then execute the following command:
[source,bash]
----
[[pve_ceph_manager]]
Ceph Manager
------------
+
The Manager daemon runs alongside the monitors. It provides an interface to
-monitor the cluster. Since the Ceph luminous release at least one ceph-mgr
-footnote:[Ceph Manager https://docs.ceph.com/docs/{ceph_codename}/mgr/] daemon is
+monitor the cluster. Since the release of Ceph luminous, at least one ceph-mgr
+footnote:[Ceph Manager {cephdocs-url}/mgr/] daemon is
required.
[[pveceph_create_mgr]]
Create Manager
~~~~~~~~~~~~~~
-Multiple Managers can be installed, but at any time only one Manager is active.
+Multiple Managers can be installed, but only one Manager is active at any given
+time.
[source,bash]
----
Destroy Manager
~~~~~~~~~~~~~~~
-To remove a Ceph Manager via the GUI first select a node in the tree view and
+To remove a Ceph Manager via the GUI, first select a node in the tree view and
go to the **Ceph -> Monitor** panel. Select the Manager and click the
**Destroy** button.
-To remove a Ceph Monitor via the CLI first connect to the node on which the
+To remove a Ceph Monitor via the CLI, first connect to the node on which the
Manager is running. Then execute the following command:
[source,bash]
----
pveceph mgr destroy
----
-NOTE: A Ceph cluster can function without a Manager, but certain functions like
-the cluster status or usage require a running Manager.
-
+NOTE: While a manager is not a hard-dependency, it is crucial for a Ceph cluster,
+as it handles important features like PG-autoscaling, device health monitoring,
+telemetry and more.
[[pve_ceph_osds]]
Ceph OSDs
---------
-Ceph **O**bject **S**torage **D**aemons are storing objects for Ceph over the
-network. It is recommended to use one OSD per physical disk.
-NOTE: By default an object is 4 MiB in size.
+[thumbnail="screenshot/gui-ceph-osd-status.png"]
+
+Ceph **O**bject **S**torage **D**aemons store objects for Ceph over the
+network. It is recommended to use one OSD per physical disk.
[[pve_ceph_osd_create]]
Create OSDs
~~~~~~~~~~~
-[thumbnail="screenshot/gui-ceph-osd-status.png"]
-
-via GUI or via CLI as follows:
+You can create an OSD either via the {pve} web interface or via the CLI using
+`pveceph`. For example:
[source,bash]
----
pveceph osd create /dev/sd[X]
----
-TIP: We recommend a Ceph cluster size, starting with 12 OSDs, distributed
-evenly among your, at least three nodes (4 OSDs on each node).
+TIP: We recommend a Ceph cluster with at least three nodes and at least 12
+OSDs, evenly distributed among the nodes.
-If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot
-sector and any OSD leftover the following command should be sufficient.
+If the disk was in use before (for example, for ZFS or as an OSD) you first need
+to zap all traces of that usage. To remove the partition table, boot sector and
+any other OSD leftover, you can use the following command:
[source,bash]
----
ceph-volume lvm zap /dev/sd[X] --destroy
----
-WARNING: The above command will destroy data on the disk!
+WARNING: The above command will destroy all data on the disk!
.Ceph Bluestore
Starting with the Ceph Kraken release, a new Ceph OSD storage type was
-introduced, the so called Bluestore
+introduced called Bluestore
footnote:[Ceph Bluestore https://ceph.com/community/new-luminous-bluestore/].
This is the default when creating OSDs since Ceph Luminous.
pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]
----
-You can directly choose the size for those with the '-db_size' and '-wal_size'
-parameters respectively. If they are not given the following values (in order)
+You can directly choose the size of those with the '-db_size' and '-wal_size'
+parameters respectively. If they are not given, the following values (in order)
will be used:
-* bluestore_block_{db,wal}_size from ceph configuration...
+* bluestore_block_{db,wal}_size from Ceph configuration...
** ... database, section 'osd'
** ... database, section 'global'
** ... file, section 'osd'
** ... file, section 'global'
* 10% (DB)/1% (WAL) of OSD size
-NOTE: The DB stores BlueStore’s internal metadata and the WAL is BlueStore’s
+NOTE: The DB stores BlueStore’s internal metadata, and the WAL is BlueStore’s
internal journal or write-ahead log. It is recommended to use a fast SSD or
NVRAM for better performance.
-
.Ceph Filestore
-Before Ceph Luminous, Filestore was used as default storage type for Ceph OSDs.
+Before Ceph Luminous, Filestore was used as the default storage type for Ceph OSDs.
Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
'pveceph' anymore. If you still want to create filestore OSDs, use
'ceph-volume' directly.
Destroy OSDs
~~~~~~~~~~~~
-To remove an OSD via the GUI first select a {PVE} node in the tree view and go
-to the **Ceph -> OSD** panel. Select the OSD to destroy. Next click the **OUT**
-button. Once the OSD status changed from `in` to `out` click the **STOP**
-button. As soon as the status changed from `up` to `down` select **Destroy**
-from the `More` drop-down menu.
+To remove an OSD via the GUI, first select a {PVE} node in the tree view and go
+to the **Ceph -> OSD** panel. Then select the OSD to destroy and click the **OUT**
+button. Once the OSD status has changed from `in` to `out`, click the **STOP**
+button. Finally, after the status has changed from `up` to `down`, select
+**Destroy** from the `More` drop-down menu.
To remove an OSD via the CLI run the following commands.
+
[source,bash]
----
ceph osd out <ID>
systemctl stop ceph-osd@<ID>.service
----
+
NOTE: The first command instructs Ceph not to include the OSD in the data
distribution. The second command stops the OSD service. Until this time, no
data is lost.
The following command destroys the OSD. Specify the '-cleanup' option to
additionally destroy the partition table.
+
[source,bash]
----
pveceph osd destroy <ID>
----
-WARNING: The above command will destroy data on the disk!
+
+WARNING: The above command will destroy all data on the disk!
[[pve_ceph_pools]]
Ceph Pools
----------
-A pool is a logical group for storing objects. It holds **P**lacement
-**G**roups (`PG`, `pg_num`), a collection of objects.
-
-
-Create Pools
-~~~~~~~~~~~~
[thumbnail="screenshot/gui-ceph-pools.png"]
-When no options are given, we set a default of **128 PGs**, a **size of 3
-replicas** and a **min_size of 2 replicas** for serving objects in a degraded
-state.
+A pool is a logical group for storing objects. It holds a collection of objects,
+known as **P**lacement **G**roups (`PG`, `pg_num`).
-NOTE: The default number of PGs works for 2-5 disks. Ceph throws a
-'HEALTH_WARNING' if you have too few or too many PGs in your cluster.
-
-It is advised to calculate the PG number depending on your setup, you can find
-the formula and the PG calculator footnote:[PG calculator
-https://ceph.com/pgcalc/] online. While PGs can be increased later on, they can
-never be decreased.
+Create and Edit Pools
+~~~~~~~~~~~~~~~~~~~~~
-You can create pools through command line or on the GUI on each PVE host under
-**Ceph -> Pools**.
+You can create and edit pools from the command line or the web interface of any
+{pve} host under **Ceph -> Pools**.
+When no options are given, we set a default of **128 PGs**, a **size of 3
+replicas** and a **min_size of 2 replicas**, to ensure no data loss occurs if
+any OSD fails.
+
+WARNING: **Do not set a min_size of 1**. A replicated pool with min_size of 1
+allows I/O on an object when it has only 1 replica, which could lead to data
+loss, incomplete PGs or unfound objects.
+
+It is advised that you either enable the PG-Autoscaler or calculate the PG
+number based on your setup. You can find the formula and the PG calculator
+footnote:[PG calculator https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/] online. From Ceph Nautilus
+onward, you can change the number of PGs
+footnoteref:[placement_groups,Placement Groups
+{cephdocs-url}/rados/operations/placement-groups/] after the setup.
+
+The PG autoscaler footnoteref:[autoscaler,Automated Scaling
+{cephdocs-url}/rados/operations/placement-groups/#automated-scaling] can
+automatically scale the PG count for a pool in the background. Setting the
+`Target Size` or `Target Ratio` advanced parameters helps the PG-Autoscaler to
+make better decisions.
+
+.Example for creating a pool over the CLI
[source,bash]
----
-pveceph pool create <name>
+pveceph pool create <pool-name> --add_storages
----
-If you would like to automatically also get a storage definition for your pool,
-mark the checkbox "Add storages" in the GUI or use the command line option
-'--add_storages' at pool creation.
+TIP: If you would also like to automatically define a storage for your
+pool, keep the `Add as Storage' checkbox checked in the web interface, or use the
+command-line option '--add_storages' at pool creation.
+
+Pool Options
+^^^^^^^^^^^^
+
+[thumbnail="screenshot/gui-ceph-pool-create.png"]
+
+The following options are available on pool creation, and partially also when
+editing a pool.
+
+Name:: The name of the pool. This must be unique and can't be changed afterwards.
+Size:: The number of replicas per object. Ceph always tries to have this many
+copies of an object. Default: `3`.
+PG Autoscale Mode:: The automatic PG scaling mode footnoteref:[autoscaler] of
+the pool. If set to `warn`, it produces a warning message when a pool
+has a non-optimal PG count. Default: `warn`.
+Add as Storage:: Configure a VM or container storage using the new pool.
+Default: `true` (only visible on creation).
+
+.Advanced Options
+Min. Size:: The minimum number of replicas per object. Ceph will reject I/O on
+the pool if a PG has less than this many replicas. Default: `2`.
+Crush Rule:: The rule to use for mapping object placement in the cluster. These
+rules define how data is placed within the cluster. See
+xref:pve_ceph_device_classes[Ceph CRUSH & device classes] for information on
+device-based rules.
+# of PGs:: The number of placement groups footnoteref:[placement_groups] that
+the pool should have at the beginning. Default: `128`.
+Target Ratio:: The ratio of data that is expected in the pool. The PG
+autoscaler uses the ratio relative to other ratio sets. It takes precedence
+over the `target size` if both are set.
+Target Size:: The estimated amount of data expected in the pool. The PG
+autoscaler uses this size to estimate the optimal PG count.
+Min. # of PGs:: The minimum number of placement groups. This setting is used to
+fine-tune the lower bound of the PG count for that pool. The PG autoscaler
+will not merge PGs below this threshold.
Further information on Ceph pool handling can be found in the Ceph pool
operation footnote:[Ceph pool operation
-https://docs.ceph.com/docs/{ceph_codename}/rados/operations/pools/]
+{cephdocs-url}/rados/operations/pools/]
manual.
+[[pve_ceph_ec_pools]]
+Erasure Coded Pools
+~~~~~~~~~~~~~~~~~~~
+
+Erasure coding (EC) is a form of `forward error correction' codes that allows
+to recover from a certain amount of data loss. Erasure coded pools can offer
+more usable space compared to replicated pools, but they do that for the price
+of performance.
+
+For comparison: in classic, replicated pools, multiple replicas of the data
+are stored (`size`) while in erasure coded pool, data is split into `k` data
+chunks with additional `m` coding (checking) chunks. Those coding chunks can be
+used to recreate data should data chunks be missing.
+
+The number of coding chunks, `m`, defines how many OSDs can be lost without
+losing any data. The total amount of objects stored is `k + m`.
+
+Creating EC Pools
+^^^^^^^^^^^^^^^^^
+
+Erasure coded (EC) pools can be created with the `pveceph` CLI tooling.
+Planning an EC pool needs to account for the fact, that they work differently
+than replicated pools.
+
+The default `min_size` of an EC pool depends on the `m` parameter. If `m = 1`,
+the `min_size` of the EC pool will be `k`. The `min_size` will be `k + 1` if
+`m > 1`. The Ceph documentation recommends a conservative `min_size` of `k + 2`
+footnote:[Ceph Erasure Coded Pool Recovery
+{cephdocs-url}/rados/operations/erasure-code/#erasure-coded-pool-recovery].
+
+If there are less than `min_size` OSDs available, any IO to the pool will be
+blocked until there are enough OSDs available again.
+
+NOTE: When planning an erasure coded pool, keep an eye on the `min_size` as it
+defines how many OSDs need to be available. Otherwise, IO will be blocked.
+
+For example, an EC pool with `k = 2` and `m = 1` will have `size = 3`,
+`min_size = 2` and will stay operational if one OSD fails. If the pool is
+configured with `k = 2`, `m = 2`, it will have a `size = 4` and `min_size = 3`
+and stay operational if one OSD is lost.
+
+To create a new EC pool, run the following command:
+
+[source,bash]
+----
+pveceph pool create <pool-name> --erasure-coding k=2,m=1
+----
+
+Optional parameters are `failure-domain` and `device-class`. If you
+need to change any EC profile settings used by the pool, you will have to
+create a new pool with a new profile.
+
+This will create a new EC pool plus the needed replicated pool to store the RBD
+omap and other metadata. In the end, there will be a `<pool name>-data` and
+`<pool name>-metada` pool. The default behavior is to create a matching storage
+configuration as well. If that behavior is not wanted, you can disable it by
+providing the `--add_storages 0` parameter. When configuring the storage
+configuration manually, keep in mind that the `data-pool` parameter needs to be
+set. Only then will the EC pool be used to store the data objects. For example:
+
+NOTE: The optional parameters `--size`, `--min_size` and `--crush_rule` will be
+used for the replicated metadata pool, but not for the erasure coded data pool.
+If you need to change the `min_size` on the data pool, you can do it later.
+The `size` and `crush_rule` parameters cannot be changed on erasure coded
+pools.
+
+If there is a need to further customize the EC profile, you can do so by
+creating it with the Ceph tools directly footnote:[Ceph Erasure Code Profile
+{cephdocs-url}/rados/operations/erasure-code/#erasure-code-profiles], and
+specify the profile to use with the `profile` parameter.
+
+For example:
+[source,bash]
+----
+pveceph pool create <pool-name> --erasure-coding profile=<profile-name>
+----
+
+Adding EC Pools as Storage
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can add an already existing EC pool as storage to {pve}. It works the same
+way as adding an `RBD` pool but requires the extra `data-pool` option.
+
+[source,bash]
+----
+pvesm add rbd <storage-name> --pool <replicated-pool> --data-pool <ec-pool>
+----
+
+TIP: Do not forget to add the `keyring` and `monhost` option for any external
+Ceph clusters, not managed by the local {pve} cluster.
+
Destroy Pools
~~~~~~~~~~~~~
-To destroy a pool via the GUI select a node in the tree view and go to the
+To destroy a pool via the GUI, select a node in the tree view and go to the
**Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**
-button. To confirm the destruction of the pool you need to enter the pool name.
+button. To confirm the destruction of the pool, you need to enter the pool name.
Run the following command to destroy a pool. Specify the '-remove_storages' to
also remove the associated storage.
+
[source,bash]
----
pveceph pool destroy <name>
----
-NOTE: Deleting the data of a pool is a background task and can take some time.
-You will notice that the data usage in the cluster is decreasing.
+NOTE: Pool deletion runs in the background and can take some time.
+You will notice the data usage in the cluster decreasing throughout this
+process.
+
+
+PG Autoscaler
+~~~~~~~~~~~~~
+
+The PG autoscaler allows the cluster to consider the amount of (expected) data
+stored in each pool and to choose the appropriate pg_num values automatically.
+It is available since Ceph Nautilus.
+
+You may need to activate the PG autoscaler module before adjustments can take
+effect.
+
+[source,bash]
+----
+ceph mgr module enable pg_autoscaler
+----
+
+The autoscaler is configured on a per pool basis and has the following modes:
+
+[horizontal]
+warn:: A health warning is issued if the suggested `pg_num` value differs too
+much from the current value.
+on:: The `pg_num` is adjusted automatically with no need for any manual
+interaction.
+off:: No automatic `pg_num` adjustments are made, and no warning will be issued
+if the PG count is not optimal.
+
+The scaling factor can be adjusted to facilitate future data storage with the
+`target_size`, `target_size_ratio` and the `pg_num_min` options.
+
+WARNING: By default, the autoscaler considers tuning the PG count of a pool if
+it is off by a factor of 3. This will lead to a considerable shift in data
+placement and might introduce a high load on the cluster.
+
+You can find a more in-depth introduction to the PG autoscaler on Ceph's Blog -
+https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/[New in
+Nautilus: PG merging and autotuning].
+
[[pve_ceph_device_classes]]
Ceph CRUSH & device classes
---------------------------
-The foundation of Ceph is its algorithm, **C**ontrolled **R**eplication
-**U**nder **S**calable **H**ashing
-(CRUSH footnote:[CRUSH https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf]).
-CRUSH calculates where to store to and retrieve data from, this has the
-advantage that no central index service is needed. CRUSH works with a map of
+[thumbnail="screenshot/gui-ceph-config.png"]
+
+The footnote:[CRUSH
+https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf] (**C**ontrolled
+**R**eplication **U**nder **S**calable **H**ashing) algorithm is at the
+foundation of Ceph.
+
+CRUSH calculates where to store and retrieve data from. This has the
+advantage that no central indexing service is needed. CRUSH works using a map of
OSDs, buckets (device locations) and rulesets (data replication) for pools.
NOTE: Further information can be found in the Ceph documentation, under the
-section CRUSH map footnote:[CRUSH map https://docs.ceph.com/docs/{ceph_codename}/rados/operations/crush-map/].
+section CRUSH map footnote:[CRUSH map {cephdocs-url}/rados/operations/crush-map/].
This map can be altered to reflect different replication hierarchies. The object
-replicas can be separated (eg. failure domains), while maintaining the desired
+replicas can be separated (e.g., failure domains), while maintaining the desired
distribution.
-A common use case is to use different classes of disks for different Ceph pools.
-For this reason, Ceph introduced the device classes with luminous, to
+A common configuration is to use different classes of disks for different Ceph
+pools. For this reason, Ceph introduced device classes with luminous, to
accommodate the need for easy ruleset generation.
The device classes can be seen in the 'ceph osd tree' output. These classes
14 nvme 0.72769 osd.14
----
-To let a pool distribute its objects only on a specific device class, you need
-to create a ruleset with the specific class first.
+To instruct a pool to only distribute objects on a specific device class, you
+first need to create a ruleset for the device class:
[source, bash]
----
[frame="none",grid="none", align="left", cols="30%,70%"]
|===
|<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
-|<root>|which crush root it should belong to (default ceph root "default")
+|<root>|which crush root it should belong to (default Ceph root "default")
|<failure-domain>|at which failure-domain the objects should be distributed (usually host)
-|<class>|what type of OSD backing store to use (eg. nvme, ssd, hdd)
+|<class>|what type of OSD backing store to use (e.g., nvme, ssd, hdd)
|===
Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
ceph osd pool set <pool-name> crush_rule <rule-name>
----
-TIP: If the pool already contains objects, all of these have to be moved
-accordingly. Depending on your setup this may introduce a big performance hit
-on your cluster. As an alternative, you can create a new pool and move disks
-separately.
+TIP: If the pool already contains objects, these must be moved accordingly.
+Depending on your setup, this may introduce a big performance impact on your
+cluster. As an alternative, you can create a new pool and move disks separately.
Ceph Client
[thumbnail="screenshot/gui-ceph-log.png"]
-You can then configure {pve} to use such pools to store VM or
-Container images. Simply use the GUI too add a new `RBD` storage (see
-section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
+Following the setup from the previous sections, you can configure {pve} to use
+such pools to store VM and Container images. Simply use the GUI to add a new
+`RBD` storage (see section
+xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
You also need to copy the keyring to a predefined location for an external Ceph
cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
done automatically.
-NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
-the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
-`my-ceph-storage` in the following example:
+NOTE: The filename needs to be `<storage_id> + `.keyring`, where `<storage_id>` is
+the expression after 'rbd:' in `/etc/pve/storage.cfg`. In the following example,
+`my-ceph-storage` is the `<storage_id>`:
[source,bash]
----
CephFS
------
-Ceph provides also a filesystem running on top of the same object storage as
-RADOS block devices do. A **M**eta**d**ata **S**erver (`MDS`) is used to map
-the RADOS backed objects to files and directories, allowing to provide a
-POSIX-compliant replicated filesystem. This allows one to have a clustered
-highly available shared filesystem in an easy way if ceph is already used. Its
-Metadata Servers guarantee that files get balanced out over the whole Ceph
-cluster, this way even high load will not overload a single host, which can be
-an issue with traditional shared filesystem approaches, like `NFS`, for
-example.
+Ceph also provides a filesystem, which runs on top of the same object storage as
+RADOS block devices do. A **M**eta**d**ata **S**erver (`MDS`) is used to map the
+RADOS backed objects to files and directories, allowing Ceph to provide a
+POSIX-compliant, replicated filesystem. This allows you to easily configure a
+clustered, highly available, shared filesystem. Ceph's Metadata Servers
+guarantee that files are evenly distributed over the entire Ceph cluster. As a
+result, even cases of high load will not overwhelm a single host, which can be
+an issue with traditional shared filesystem approaches, for example `NFS`.
[thumbnail="screenshot/gui-node-ceph-cephfs-panel.png"]
-{pve} supports both, using an existing xref:storage_cephfs[CephFS as storage]
-to save backups, ISO files or container templates and creating a
-hyper-converged CephFS itself.
+{pve} supports both creating a hyper-converged CephFS and using an existing
+xref:storage_cephfs[CephFS as storage] to save backups, ISO files, and container
+templates.
[[pveceph_fs_mds]]
Metadata Server (MDS)
~~~~~~~~~~~~~~~~~~~~~
-CephFS needs at least one Metadata Server to be configured and running to be
-able to work. One can simply create one through the {pve} web GUI's `Node ->
-CephFS` panel or on the command line with:
+CephFS needs at least one Metadata Server to be configured and running, in order
+to function. You can create an MDS through the {pve} web GUI's `Node
+-> CephFS` panel or from the command line with:
----
pveceph mds create
----
-Multiple metadata servers can be created in a cluster. But with the default
-settings only one can be active at any time. If an MDS, or its node, becomes
+Multiple metadata servers can be created in a cluster, but with the default
+settings, only one can be active at a time. If an MDS or its node becomes
unresponsive (or crashes), another `standby` MDS will get promoted to `active`.
-One can speed up the hand-over between the active and a standby MDS up by using
-the 'hotstandby' parameter option on create, or if you have already created it
+You can speed up the handover between the active and standby MDS by using
+the 'hotstandby' parameter option on creation, or if you have already created it
you may set/add:
----
mds standby replay = true
----
-in the ceph.conf respective MDS section. With this enabled, this specific MDS
-will always poll the active one, so that it can take over faster as it is in a
-`warm` state. But naturally, the active polling will cause some additional
-performance impact on your system and active `MDS`.
+in the respective MDS section of `/etc/pve/ceph.conf`. With this enabled, the
+specified MDS will remain in a `warm` state, polling the active one, so that it
+can take over faster in case of any issues.
+
+NOTE: This active polling will have an additional performance impact on your
+system and the active `MDS`.
.Multiple Active MDS
-Since Luminous (12.2.x) you can also have multiple active metadata servers
-running, but this is normally only useful for a high count on parallel clients,
-as else the `MDS` seldom is the bottleneck. If you want to set this up please
-refer to the ceph documentation. footnote:[Configuring multiple active MDS
-daemons https://docs.ceph.com/docs/{ceph_codename}/cephfs/multimds/]
+Since Luminous (12.2.x) you can have multiple active metadata servers
+running at once, but this is normally only useful if you have a high amount of
+clients running in parallel. Otherwise the `MDS` is rarely the bottleneck in a
+system. If you want to set this up, please refer to the Ceph documentation.
+footnote:[Configuring multiple active MDS daemons
+{cephdocs-url}/cephfs/multimds/]
[[pveceph_fs_create]]
Create CephFS
~~~~~~~~~~~~~
-With {pve}'s CephFS integration into you can create a CephFS easily over the
-Web GUI, the CLI or an external API interface. Some prerequisites are required
+With {pve}'s integration of CephFS, you can easily create a CephFS using the
+web interface, CLI or an external API interface. Some prerequisites are required
for this to work:
.Prerequisites for a successful CephFS setup:
-- xref:pve_ceph_install[Install Ceph packages], if this was already done some
- time ago you might want to rerun it on an up to date system to ensure that
- also all CephFS related packages get installed.
+- xref:pve_ceph_install[Install Ceph packages] - if this was already done some
+time ago, you may want to rerun it on an up-to-date system to
+ensure that all CephFS related packages get installed.
- xref:pve_ceph_monitors[Setup Monitors]
- xref:pve_ceph_monitors[Setup your OSDs]
- xref:pveceph_fs_mds[Setup at least one MDS]
-After this got all checked and done you can simply create a CephFS through
-either the Web GUI's `Node -> CephFS` panel or the command line tool `pveceph`,
-for example with:
+After this is complete, you can simply create a CephFS through
+either the Web GUI's `Node -> CephFS` panel or the command-line tool `pveceph`,
+for example:
----
pveceph fs create --pg_num 128 --add-storage
----
-This creates a CephFS named `'cephfs'' using a pool for its data named
-`'cephfs_data'' with `128` placement groups and a pool for its metadata named
-`'cephfs_metadata'' with one quarter of the data pools placement groups (`32`).
+This creates a CephFS named 'cephfs', using a pool for its data named
+'cephfs_data' with '128' placement groups and a pool for its metadata named
+'cephfs_metadata' with one quarter of the data pool's placement groups (`32`).
Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
-Ceph documentation for more information regarding a fitting placement group
-number (`pg_num`) for your setup footnote:[Ceph Placement Groups
-https://docs.ceph.com/docs/{ceph_codename}/rados/operations/placement-groups/].
-Additionally, the `'--add-storage'' parameter will add the CephFS to the {pve}
-storage configuration after it was created successfully.
+Ceph documentation for more information regarding an appropriate placement group
+number (`pg_num`) for your setup footnoteref:[placement_groups].
+Additionally, the '--add-storage' parameter will add the CephFS to the {pve}
+storage configuration after it has been created successfully.
Destroy CephFS
~~~~~~~~~~~~~~
-WARNING: Destroying a CephFS will render all its data unusable, this cannot be
+WARNING: Destroying a CephFS will render all of its data unusable. This cannot be
undone!
-If you really want to destroy an existing CephFS you first need to stop, or
-destroy, all metadata servers (`M̀DS`). You can destroy them either over the Web
-GUI or the command line interface, with:
+To completely and gracefully remove a CephFS, the following steps are
+necessary:
+* Disconnect every non-{PVE} client (e.g. unmount the CephFS in guests).
+* Disable all related CephFS {PVE} storage entries (to prevent it from being
+ automatically mounted).
+* Remove all used resources from guests (e.g. ISOs) that are on the CephFS you
+ want to destroy.
+* Unmount the CephFS storages on all cluster nodes manually with
++
----
-pveceph mds destroy NAME
+umount /mnt/pve/<STORAGE-NAME>
----
-on each {pve} node hosting a MDS daemon.
-
-Then, you can remove (destroy) CephFS by issuing a:
-
++
+Where `<STORAGE-NAME>` is the name of the CephFS storage in your {PVE}.
+
+* Now make sure that no metadata server (`MDS`) is running for that CephFS,
+ either by stopping or destroying them. This can be done through the web
+ interface or via the command-line interface, for the latter you would issue
+ the following command:
++
----
-ceph fs rm NAME --yes-i-really-mean-it
+pveceph stop --service mds.NAME
----
-on a single node hosting Ceph. After this you may want to remove the created
-data and metadata pools, this can be done either over the Web GUI or the CLI
-with:
-
++
+to stop them, or
++
+----
+pveceph mds destroy NAME
+----
++
+to destroy them.
++
+Note that standby servers will automatically be promoted to active when an
+active `MDS` is stopped or removed, so it is best to first stop all standby
+servers.
+
+* Now you can destroy the CephFS with
++
----
-pveceph pool destroy NAME
+pveceph fs destroy NAME --remove-storages --remove-pools
----
++
+This will automatically destroy the underlying Ceph pools as well as remove
+the storages from pve config.
+After these steps, the CephFS should be completely removed and if you have
+other CephFS instances, the stopped metadata servers can be started again
+to act as standbys.
Ceph maintenance
----------------
Replace OSDs
~~~~~~~~~~~~
-One of the common maintenance tasks in Ceph is to replace a disk of an OSD. If
-a disk is already in a failed state, then you can go ahead and run through the
-steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate those
-copies on the remaining OSDs if possible. This rebalancing will start as soon
-as an OSD failure is detected or an OSD was actively stopped.
+One of the most common maintenance tasks in Ceph is to replace the disk of an
+OSD. If a disk is already in a failed state, then you can go ahead and run
+through the steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate
+those copies on the remaining OSDs if possible. This rebalancing will start as
+soon as an OSD failure is detected or an OSD was actively stopped.
NOTE: With the default size/min_size (3/2) of a pool, recovery only starts when
`size + 1` nodes are available. The reason for this is that the Ceph object
balancer xref:pve_ceph_device_classes[CRUSH] defaults to a full node as
`failure domain'.
-To replace a still functioning disk, on the GUI go through the steps in
+To replace a functioning disk from the GUI, go through the steps in
xref:pve_ceph_osd_destroy[Destroy OSDs]. The only addition is to wait until
the cluster shows 'HEALTH_OK' before stopping the OSD to destroy it.
-On the command line use the following commands.
+On the command line, use the following commands:
+
----
ceph osd out osd.<id>
----
You can check with the command below if the OSD can be safely removed.
+
----
ceph osd safe-to-destroy osd.<id>
----
-Once the above check tells you that it is save to remove the OSD, you can
-continue with following commands.
+Once the above check tells you that it is safe to remove the OSD, you can
+continue with the following commands:
+
----
systemctl stop ceph-osd@<id>.service
pveceph osd destroy <id>
Trim/Discard
~~~~~~~~~~~~
-It is a good measure to run 'fstrim' (discard) regularly on VMs or containers.
+
+It is good practice to run 'fstrim' (discard) regularly on VMs and containers.
This releases data blocks that the filesystem isn’t using anymore. It reduces
data usage and resource load. Most modern operating systems issue such discard
commands to their disks regularly. You only need to ensure that the Virtual
[[pveceph_scrub]]
Scrub & Deep Scrub
~~~~~~~~~~~~~~~~~~
+
Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every
object in a PG for its health. There are two forms of Scrubbing, daily
cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
the objects and uses checksums to ensure data integrity. If a running scrub
interferes with business (performance) needs, you can adjust the time when
-scrubs footnote:[Ceph scrubbing https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing]
+scrubs footnote:[Ceph scrubbing {cephdocs-url}/rados/configuration/osd-config-ref/#scrubbing]
are executed.
-Ceph monitoring and troubleshooting
+Ceph Monitoring and Troubleshooting
-----------------------------------
-A good start is to continuosly monitor the ceph health from the start of
-initial deployment. Either through the ceph tools itself, but also by accessing
+
+It is important to continuously monitor the health of a Ceph deployment from the
+beginning, either by using the Ceph tools or by accessing
the status through the {pve} link:api-viewer/index.html[API].
-The following ceph commands below can be used to see if the cluster is healthy
+The following Ceph commands can be used to see if the cluster is healthy
('HEALTH_OK'), if there are warnings ('HEALTH_WARN'), or even errors
-('HEALTH_ERR'). If the cluster is in an unhealthy state the status commands
+('HEALTH_ERR'). If the cluster is in an unhealthy state, the status commands
below will also give you an overview of the current events and actions to take.
----
pve# ceph -w
----
-To get a more detailed view, every ceph service has a log file under
-`/var/log/ceph/` and if there is not enough detail, the log level can be
-adjusted footnote:[Ceph log and debugging https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/log-and-debug/].
+To get a more detailed view, every Ceph service has a log file under
+`/var/log/ceph/`. If more detail is required, the log level can be
+adjusted footnote:[Ceph log and debugging {cephdocs-url}/rados/troubleshooting/log-and-debug/].
You can find more information about troubleshooting
-footnote:[Ceph troubleshooting https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/]
+footnote:[Ceph troubleshooting {cephdocs-url}/rados/troubleshooting/]
a Ceph cluster on the official website.
projects / pve-docs.git / blobdiff