10 pveceph - Manage Ceph Services on Proxmox VE Nodes
15 include::pveceph.1-synopsis.adoc[]
21 Deploy Hyper-Converged Ceph Cluster
22 ===================================
26 [thumbnail="screenshot/gui-ceph-status.png"]
28 {pve} unifies your compute and storage systems, i.e. you can use the same
29 physical nodes within a cluster for both computing (processing VMs and
30 containers) and replicated storage. The traditional silos of compute and
31 storage resources can be wrapped up into a single hyper-converged appliance.
32 Separate storage networks (SANs) and connections via network attached storages
33 (NAS) disappear. With the integration of Ceph, an open source software-defined
34 storage platform, {pve} has the ability to run and manage Ceph storage directly
35 on the hypervisor nodes.
37 Ceph is a distributed object store and file system designed to provide
38 excellent performance, reliability and scalability.
40 .Some advantages of Ceph on {pve} are:
41 - Easy setup and management with CLI and GUI support
45 - Scalable to the exabyte level
46 - Setup pools with different performance and redundancy characteristics
47 - Data is replicated, making it fault tolerant
48 - Runs on economical commodity hardware
49 - No need for hardware RAID controllers
52 For small to mid sized deployments, it is possible to install a Ceph server for
53 RADOS Block Devices (RBD) directly on your {pve} cluster nodes, see
54 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
55 hardware has plenty of CPU power and RAM, so running storage services
56 and VMs on the same node is possible.
58 To simplify management, we provide 'pveceph' - a tool to install and
59 manage {ceph} services on {pve} nodes.
61 .Ceph consists of a couple of Daemons footnote:[Ceph intro https://docs.ceph.com/docs/{ceph_codename}/start/intro/], for use as a RBD storage:
62 - Ceph Monitor (ceph-mon)
63 - Ceph Manager (ceph-mgr)
64 - Ceph OSD (ceph-osd; Object Storage Daemon)
66 TIP: We highly recommend to get familiar with Ceph's architecture
67 footnote:[Ceph architecture https://docs.ceph.com/docs/{ceph_codename}/architecture/]
69 footnote:[Ceph glossary https://docs.ceph.com/docs/{ceph_codename}/glossary].
75 To build a hyper-converged Proxmox + Ceph Cluster there should be at least
76 three (preferably) identical servers for the setup.
78 Check also the recommendations from
79 https://docs.ceph.com/docs/{ceph_codename}/start/hardware-recommendations/[Ceph's website].
82 Higher CPU core frequency reduce latency and should be preferred. As a simple
83 rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
84 provide enough resources for stable and durable Ceph performance.
87 Especially in a hyper-converged setup, the memory consumption needs to be
88 carefully monitored. In addition to the intended workload from virtual machines
89 and container, Ceph needs enough memory available to provide good and stable
90 performance. As a rule of thumb, for roughly 1 TiB of data, 1 GiB of memory
91 will be used by an OSD. OSD caching will use additional memory.
94 We recommend a network bandwidth of at least 10 GbE or more, which is used
95 exclusively for Ceph. A meshed network setup
96 footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
97 is also an option if there are no 10 GbE switches available.
99 The volume of traffic, especially during recovery, will interfere with other
100 services on the same network and may even break the {pve} cluster stack.
102 Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
103 link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate
104 10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwidth
105 will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or
106 even 100 GBps are possible.
109 When planning the size of your Ceph cluster, it is important to take the
110 recovery time into consideration. Especially with small clusters, the recovery
111 might take long. It is recommended that you use SSDs instead of HDDs in small
112 setups to reduce recovery time, minimizing the likelihood of a subsequent
113 failure event during recovery.
115 In general SSDs will provide more IOPs than spinning disks. This fact and the
116 higher cost may make a xref:pve_ceph_device_classes[class based] separation of
117 pools appealing. Another possibility to speedup OSDs is to use a faster disk
118 as journal or DB/**W**rite-**A**head-**L**og device, see
119 xref:pve_ceph_osds[creating Ceph OSDs]. If a faster disk is used for multiple
120 OSDs, a proper balance between OSD and WAL / DB (or journal) disk must be
121 selected, otherwise the faster disk becomes the bottleneck for all linked OSDs.
123 Aside from the disk type, Ceph best performs with an even sized and distributed
124 amount of disks per node. For example, 4 x 500 GB disks with in each node is
125 better than a mixed setup with a single 1 TB and three 250 GB disk.
127 One also need to balance OSD count and single OSD capacity. More capacity
128 allows to increase storage density, but it also means that a single OSD
129 failure forces ceph to recover more data at once.
132 As Ceph handles data object redundancy and multiple parallel writes to disks
133 (OSDs) on its own, using a RAID controller normally doesn’t improve
134 performance or availability. On the contrary, Ceph is designed to handle whole
135 disks on it's own, without any abstraction in between. RAID controller are not
136 designed for the Ceph use case and may complicate things and sometimes even
137 reduce performance, as their write and caching algorithms may interfere with
140 WARNING: Avoid RAID controller, use host bus adapter (HBA) instead.
142 NOTE: Above recommendations should be seen as a rough guidance for choosing
143 hardware. Therefore, it is still essential to adapt it to your specific needs,
144 test your setup and monitor health and performance continuously.
146 [[pve_ceph_install_wizard]]
147 Initial Ceph installation & configuration
148 -----------------------------------------
150 [thumbnail="screenshot/gui-node-ceph-install.png"]
152 With {pve} you have the benefit of an easy to use installation wizard
153 for Ceph. Click on one of your cluster nodes and navigate to the Ceph
154 section in the menu tree. If Ceph is not already installed you will be
155 offered to do so now.
157 The wizard is divided into different sections, where each needs to be
158 finished successfully in order to use Ceph. After starting the installation
159 the wizard will download and install all required packages from {pve}'s ceph
162 After finishing the first step, you will need to create a configuration.
163 This step is only needed once per cluster, as this configuration is distributed
164 automatically to all remaining cluster members through {pve}'s clustered
165 xref:chapter_pmxcfs[configuration file system (pmxcfs)].
167 The configuration step includes the following settings:
169 * *Public Network:* You should setup a dedicated network for Ceph, this
170 setting is required. Separating your Ceph traffic is highly recommended,
171 because it could lead to troubles with other latency dependent services,
172 e.g., cluster communication may decrease Ceph's performance, if not done.
174 [thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
176 * *Cluster Network:* As an optional step you can go even further and
177 separate the xref:pve_ceph_osds[OSD] replication & heartbeat traffic
178 as well. This will relieve the public network and could lead to
179 significant performance improvements especially in big clusters.
181 You have two more options which are considered advanced and therefore
182 should only changed if you are an expert.
184 * *Number of replicas*: Defines the how often a object is replicated
185 * *Minimum replicas*: Defines the minimum number of required replicas
186 for I/O to be marked as complete.
188 Additionally you need to choose your first monitor node, this is required.
190 That's it, you should see a success page as the last step with further
191 instructions on how to go on. You are now prepared to start using Ceph,
192 even though you will need to create additional xref:pve_ceph_monitors[monitors],
193 create some xref:pve_ceph_osds[OSDs] and at least one xref:pve_ceph_pools[pool].
195 The rest of this chapter will guide you on how to get the most out of
196 your {pve} based Ceph setup, this will include aforementioned and
197 more like xref:pveceph_fs[CephFS] which is a very handy addition to your
201 Installation of Ceph Packages
202 -----------------------------
203 Use {pve} Ceph installation wizard (recommended) or run the following
204 command on each node:
211 This sets up an `apt` package repository in
212 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
215 Create initial Ceph configuration
216 ---------------------------------
218 [thumbnail="screenshot/gui-ceph-config.png"]
220 Use the {pve} Ceph installation wizard (recommended) or run the
221 following command on one node:
225 pveceph init --network 10.10.10.0/24
228 This creates an initial configuration at `/etc/pve/ceph.conf` with a
229 dedicated network for ceph. That file is automatically distributed to
230 all {pve} nodes by using xref:chapter_pmxcfs[pmxcfs]. The command also
231 creates a symbolic link from `/etc/ceph/ceph.conf` pointing to that file.
232 So you can simply run Ceph commands without the need to specify a
236 [[pve_ceph_monitors]]
239 The Ceph Monitor (MON)
240 footnote:[Ceph Monitor https://docs.ceph.com/docs/{ceph_codename}/start/intro/]
241 maintains a master copy of the cluster map. For high availability you need to
242 have at least 3 monitors. One monitor will already be installed if you
243 used the installation wizard. You won't need more than 3 monitors as long
244 as your cluster is small to midsize, only really large clusters will
248 [[pveceph_create_mon]]
252 [thumbnail="screenshot/gui-ceph-monitor.png"]
254 On each node where you want to place a monitor (three monitors are recommended),
255 create it by using the 'Ceph -> Monitor' tab in the GUI or run.
263 [[pveceph_destroy_mon]]
267 To remove a Ceph Monitor via the GUI first select a node in the tree view and
268 go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**
271 To remove a Ceph Monitor via the CLI first connect to the node on which the MON
272 is running. Then execute the following command:
278 NOTE: At least three Monitors are needed for quorum.
284 The Manager daemon runs alongside the monitors. It provides an interface to
285 monitor the cluster. Since the Ceph luminous release at least one ceph-mgr
286 footnote:[Ceph Manager https://docs.ceph.com/docs/{ceph_codename}/mgr/] daemon is
289 [[pveceph_create_mgr]]
293 Multiple Managers can be installed, but at any time only one Manager is active.
300 NOTE: It is recommended to install the Ceph Manager on the monitor nodes. For
301 high availability install more then one manager.
304 [[pveceph_destroy_mgr]]
308 To remove a Ceph Manager via the GUI first select a node in the tree view and
309 go to the **Ceph -> Monitor** panel. Select the Manager and click the
312 To remove a Ceph Monitor via the CLI first connect to the node on which the
313 Manager is running. Then execute the following command:
319 NOTE: A Ceph cluster can function without a Manager, but certain functions like
320 the cluster status or usage require a running Manager.
326 Ceph **O**bject **S**torage **D**aemons are storing objects for Ceph over the
327 network. It is recommended to use one OSD per physical disk.
329 NOTE: By default an object is 4 MiB in size.
331 [[pve_ceph_osd_create]]
335 [thumbnail="screenshot/gui-ceph-osd-status.png"]
337 via GUI or via CLI as follows:
341 pveceph osd create /dev/sd[X]
344 TIP: We recommend a Ceph cluster size, starting with 12 OSDs, distributed
345 evenly among your, at least three nodes (4 OSDs on each node).
347 If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot
348 sector and any OSD leftover the following command should be sufficient.
352 ceph-volume lvm zap /dev/sd[X] --destroy
355 WARNING: The above command will destroy data on the disk!
359 Starting with the Ceph Kraken release, a new Ceph OSD storage type was
360 introduced, the so called Bluestore
361 footnote:[Ceph Bluestore https://ceph.com/community/new-luminous-bluestore/].
362 This is the default when creating OSDs since Ceph Luminous.
366 pveceph osd create /dev/sd[X]
369 .Block.db and block.wal
371 If you want to use a separate DB/WAL device for your OSDs, you can specify it
372 through the '-db_dev' and '-wal_dev' options. The WAL is placed with the DB, if
373 not specified separately.
377 pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]
380 You can directly choose the size for those with the '-db_size' and '-wal_size'
381 parameters respectively. If they are not given the following values (in order)
384 * bluestore_block_{db,wal}_size from ceph configuration...
385 ** ... database, section 'osd'
386 ** ... database, section 'global'
387 ** ... file, section 'osd'
388 ** ... file, section 'global'
389 * 10% (DB)/1% (WAL) of OSD size
391 NOTE: The DB stores BlueStore’s internal metadata and the WAL is BlueStore’s
392 internal journal or write-ahead log. It is recommended to use a fast SSD or
393 NVRAM for better performance.
398 Before Ceph Luminous, Filestore was used as default storage type for Ceph OSDs.
399 Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
400 'pveceph' anymore. If you still want to create filestore OSDs, use
401 'ceph-volume' directly.
405 ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
408 [[pve_ceph_osd_destroy]]
412 To remove an OSD via the GUI first select a {PVE} node in the tree view and go
413 to the **Ceph -> OSD** panel. Select the OSD to destroy. Next click the **OUT**
414 button. Once the OSD status changed from `in` to `out` click the **STOP**
415 button. As soon as the status changed from `up` to `down` select **Destroy**
416 from the `More` drop-down menu.
418 To remove an OSD via the CLI run the following commands.
422 systemctl stop ceph-osd@<ID>.service
424 NOTE: The first command instructs Ceph not to include the OSD in the data
425 distribution. The second command stops the OSD service. Until this time, no
428 The following command destroys the OSD. Specify the '-cleanup' option to
429 additionally destroy the partition table.
432 pveceph osd destroy <ID>
434 WARNING: The above command will destroy data on the disk!
440 A pool is a logical group for storing objects. It holds **P**lacement
441 **G**roups (`PG`, `pg_num`), a collection of objects.
447 [thumbnail="screenshot/gui-ceph-pools.png"]
449 When no options are given, we set a default of **128 PGs**, a **size of 3
450 replicas** and a **min_size of 2 replicas** for serving objects in a degraded
453 NOTE: The default number of PGs works for 2-5 disks. Ceph throws a
454 'HEALTH_WARNING' if you have too few or too many PGs in your cluster.
456 It is advised to calculate the PG number depending on your setup, you can find
457 the formula and the PG calculator footnote:[PG calculator
458 https://ceph.com/pgcalc/] online. While PGs can be increased later on, they can
462 You can create pools through command line or on the GUI on each PVE host under
467 pveceph pool create <name>
470 If you would like to automatically also get a storage definition for your pool,
471 mark the checkbox "Add storages" in the GUI or use the command line option
472 '--add_storages' at pool creation.
474 Further information on Ceph pool handling can be found in the Ceph pool
475 operation footnote:[Ceph pool operation
476 https://docs.ceph.com/docs/{ceph_codename}/rados/operations/pools/]
483 To destroy a pool via the GUI select a node in the tree view and go to the
484 **Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**
485 button. To confirm the destruction of the pool you need to enter the pool name.
487 Run the following command to destroy a pool. Specify the '-remove_storages' to
488 also remove the associated storage.
491 pveceph pool destroy <name>
494 NOTE: Deleting the data of a pool is a background task and can take some time.
495 You will notice that the data usage in the cluster is decreasing.
497 [[pve_ceph_device_classes]]
498 Ceph CRUSH & device classes
499 ---------------------------
500 The foundation of Ceph is its algorithm, **C**ontrolled **R**eplication
501 **U**nder **S**calable **H**ashing
502 (CRUSH footnote:[CRUSH https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf]).
504 CRUSH calculates where to store to and retrieve data from, this has the
505 advantage that no central index service is needed. CRUSH works with a map of
506 OSDs, buckets (device locations) and rulesets (data replication) for pools.
508 NOTE: Further information can be found in the Ceph documentation, under the
509 section CRUSH map footnote:[CRUSH map https://docs.ceph.com/docs/{ceph_codename}/rados/operations/crush-map/].
511 This map can be altered to reflect different replication hierarchies. The object
512 replicas can be separated (eg. failure domains), while maintaining the desired
515 A common use case is to use different classes of disks for different Ceph pools.
516 For this reason, Ceph introduced the device classes with luminous, to
517 accommodate the need for easy ruleset generation.
519 The device classes can be seen in the 'ceph osd tree' output. These classes
520 represent their own root bucket, which can be seen with the below command.
524 ceph osd crush tree --show-shadow
527 Example output form the above command:
531 ID CLASS WEIGHT TYPE NAME
532 -16 nvme 2.18307 root default~nvme
533 -13 nvme 0.72769 host sumi1~nvme
534 12 nvme 0.72769 osd.12
535 -14 nvme 0.72769 host sumi2~nvme
536 13 nvme 0.72769 osd.13
537 -15 nvme 0.72769 host sumi3~nvme
538 14 nvme 0.72769 osd.14
539 -1 7.70544 root default
540 -3 2.56848 host sumi1
541 12 nvme 0.72769 osd.12
542 -5 2.56848 host sumi2
543 13 nvme 0.72769 osd.13
544 -7 2.56848 host sumi3
545 14 nvme 0.72769 osd.14
548 To let a pool distribute its objects only on a specific device class, you need
549 to create a ruleset with the specific class first.
553 ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> <class>
556 [frame="none",grid="none", align="left", cols="30%,70%"]
558 |<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
559 |<root>|which crush root it should belong to (default ceph root "default")
560 |<failure-domain>|at which failure-domain the objects should be distributed (usually host)
561 |<class>|what type of OSD backing store to use (eg. nvme, ssd, hdd)
564 Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
568 ceph osd pool set <pool-name> crush_rule <rule-name>
571 TIP: If the pool already contains objects, all of these have to be moved
572 accordingly. Depending on your setup this may introduce a big performance hit
573 on your cluster. As an alternative, you can create a new pool and move disks
580 [thumbnail="screenshot/gui-ceph-log.png"]
582 You can then configure {pve} to use such pools to store VM or
583 Container images. Simply use the GUI too add a new `RBD` storage (see
584 section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
586 You also need to copy the keyring to a predefined location for an external Ceph
587 cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
590 NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
591 the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
592 `my-ceph-storage` in the following example:
596 mkdir /etc/pve/priv/ceph
597 cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
604 Ceph provides also a filesystem running on top of the same object storage as
605 RADOS block devices do. A **M**eta**d**ata **S**erver (`MDS`) is used to map
606 the RADOS backed objects to files and directories, allowing to provide a
607 POSIX-compliant replicated filesystem. This allows one to have a clustered
608 highly available shared filesystem in an easy way if ceph is already used. Its
609 Metadata Servers guarantee that files get balanced out over the whole Ceph
610 cluster, this way even high load will not overload a single host, which can be
611 an issue with traditional shared filesystem approaches, like `NFS`, for
614 [thumbnail="screenshot/gui-node-ceph-cephfs-panel.png"]
616 {pve} supports both, using an existing xref:storage_cephfs[CephFS as storage]
617 to save backups, ISO files or container templates and creating a
618 hyper-converged CephFS itself.
622 Metadata Server (MDS)
623 ~~~~~~~~~~~~~~~~~~~~~
625 CephFS needs at least one Metadata Server to be configured and running to be
626 able to work. One can simply create one through the {pve} web GUI's `Node ->
627 CephFS` panel or on the command line with:
633 Multiple metadata servers can be created in a cluster. But with the default
634 settings only one can be active at any time. If an MDS, or its node, becomes
635 unresponsive (or crashes), another `standby` MDS will get promoted to `active`.
636 One can speed up the hand-over between the active and a standby MDS up by using
637 the 'hotstandby' parameter option on create, or if you have already created it
641 mds standby replay = true
644 in the ceph.conf respective MDS section. With this enabled, this specific MDS
645 will always poll the active one, so that it can take over faster as it is in a
646 `warm` state. But naturally, the active polling will cause some additional
647 performance impact on your system and active `MDS`.
651 Since Luminous (12.2.x) you can also have multiple active metadata servers
652 running, but this is normally only useful for a high count on parallel clients,
653 as else the `MDS` seldom is the bottleneck. If you want to set this up please
654 refer to the ceph documentation. footnote:[Configuring multiple active MDS
655 daemons https://docs.ceph.com/docs/{ceph_codename}/cephfs/multimds/]
657 [[pveceph_fs_create]]
661 With {pve}'s CephFS integration into you can create a CephFS easily over the
662 Web GUI, the CLI or an external API interface. Some prerequisites are required
665 .Prerequisites for a successful CephFS setup:
666 - xref:pve_ceph_install[Install Ceph packages], if this was already done some
667 time ago you might want to rerun it on an up to date system to ensure that
668 also all CephFS related packages get installed.
669 - xref:pve_ceph_monitors[Setup Monitors]
670 - xref:pve_ceph_monitors[Setup your OSDs]
671 - xref:pveceph_fs_mds[Setup at least one MDS]
673 After this got all checked and done you can simply create a CephFS through
674 either the Web GUI's `Node -> CephFS` panel or the command line tool `pveceph`,
678 pveceph fs create --pg_num 128 --add-storage
681 This creates a CephFS named `'cephfs'' using a pool for its data named
682 `'cephfs_data'' with `128` placement groups and a pool for its metadata named
683 `'cephfs_metadata'' with one quarter of the data pools placement groups (`32`).
684 Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
685 Ceph documentation for more information regarding a fitting placement group
686 number (`pg_num`) for your setup footnote:[Ceph Placement Groups
687 https://docs.ceph.com/docs/{ceph_codename}/rados/operations/placement-groups/].
688 Additionally, the `'--add-storage'' parameter will add the CephFS to the {pve}
689 storage configuration after it was created successfully.
694 WARNING: Destroying a CephFS will render all its data unusable, this cannot be
697 If you really want to destroy an existing CephFS you first need to stop, or
698 destroy, all metadata servers (`M̀DS`). You can destroy them either over the Web
699 GUI or the command line interface, with:
702 pveceph mds destroy NAME
704 on each {pve} node hosting a MDS daemon.
706 Then, you can remove (destroy) CephFS by issuing a:
709 ceph fs rm NAME --yes-i-really-mean-it
711 on a single node hosting Ceph. After this you may want to remove the created
712 data and metadata pools, this can be done either over the Web GUI or the CLI
716 pveceph pool destroy NAME
726 One of the common maintenance tasks in Ceph is to replace a disk of an OSD. If
727 a disk is already in a failed state, then you can go ahead and run through the
728 steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate those
729 copies on the remaining OSDs if possible. This rebalancing will start as soon
730 as an OSD failure is detected or an OSD was actively stopped.
732 NOTE: With the default size/min_size (3/2) of a pool, recovery only starts when
733 `size + 1` nodes are available. The reason for this is that the Ceph object
734 balancer xref:pve_ceph_device_classes[CRUSH] defaults to a full node as
737 To replace a still functioning disk, on the GUI go through the steps in
738 xref:pve_ceph_osd_destroy[Destroy OSDs]. The only addition is to wait until
739 the cluster shows 'HEALTH_OK' before stopping the OSD to destroy it.
741 On the command line use the following commands.
743 ceph osd out osd.<id>
746 You can check with the command below if the OSD can be safely removed.
748 ceph osd safe-to-destroy osd.<id>
751 Once the above check tells you that it is save to remove the OSD, you can
752 continue with following commands.
754 systemctl stop ceph-osd@<id>.service
755 pveceph osd destroy <id>
758 Replace the old disk with the new one and use the same procedure as described
759 in xref:pve_ceph_osd_create[Create OSDs].
763 It is a good measure to run 'fstrim' (discard) regularly on VMs or containers.
764 This releases data blocks that the filesystem isn’t using anymore. It reduces
765 data usage and resource load. Most modern operating systems issue such discard
766 commands to their disks regularly. You only need to ensure that the Virtual
767 Machines enable the xref:qm_hard_disk_discard[disk discard option].
772 Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every
773 object in a PG for its health. There are two forms of Scrubbing, daily
774 cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
775 the objects and uses checksums to ensure data integrity. If a running scrub
776 interferes with business (performance) needs, you can adjust the time when
777 scrubs footnote:[Ceph scrubbing https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing]
781 Ceph monitoring and troubleshooting
782 -----------------------------------
783 A good start is to continuosly monitor the ceph health from the start of
784 initial deployment. Either through the ceph tools itself, but also by accessing
785 the status through the {pve} link:api-viewer/index.html[API].
787 The following ceph commands below can be used to see if the cluster is healthy
788 ('HEALTH_OK'), if there are warnings ('HEALTH_WARN'), or even errors
789 ('HEALTH_ERR'). If the cluster is in an unhealthy state the status commands
790 below will also give you an overview of the current events and actions to take.
795 # continuously output status changes (press CTRL+C to stop)
799 To get a more detailed view, every ceph service has a log file under
800 `/var/log/ceph/` and if there is not enough detail, the log level can be
801 adjusted footnote:[Ceph log and debugging https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/log-and-debug/].
803 You can find more information about troubleshooting
804 footnote:[Ceph troubleshooting https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/]
805 a Ceph cluster on the official website.
809 include::pve-copyright.adoc[]