10 pveceph - Manage Ceph Services on Proxmox VE Nodes
15 include::pveceph.1-synopsis.adoc[]
21 Deploy Hyper-Converged Ceph Cluster
22 ===================================
29 [thumbnail="screenshot/gui-ceph-status-dashboard.png"]
31 {pve} unifies your compute and storage systems, that is, you can use the same
32 physical nodes within a cluster for both computing (processing VMs and
33 containers) and replicated storage. The traditional silos of compute and
34 storage resources can be wrapped up into a single hyper-converged appliance.
35 Separate storage networks (SANs) and connections via network attached storage
36 (NAS) disappear. With the integration of Ceph, an open source software-defined
37 storage platform, {pve} has the ability to run and manage Ceph storage directly
38 on the hypervisor nodes.
40 Ceph is a distributed object store and file system designed to provide
41 excellent performance, reliability and scalability.
43 .Some advantages of Ceph on {pve} are:
44 - Easy setup and management via CLI and GUI
48 - Scalable to the exabyte level
49 - Provides block, file system, and object storage
50 - Setup pools with different performance and redundancy characteristics
51 - Data is replicated, making it fault tolerant
52 - Runs on commodity hardware
53 - No need for hardware RAID controllers
56 For small to medium-sized deployments, it is possible to install a Ceph server
57 for using RADOS Block Devices (RBD) or CephFS directly on your {pve} cluster
58 nodes (see xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
59 Recent hardware has a lot of CPU power and RAM, so running storage services and
60 virtual guests on the same node is possible.
62 To simplify management, {pve} provides you native integration to install and
63 manage {ceph} services on {pve} nodes either via the built-in web interface, or
64 using the 'pveceph' command line tool.
70 // TODO: extend and also describe basic architecture here.
71 .Ceph consists of multiple Daemons, for use as an RBD storage:
72 - Ceph Monitor (ceph-mon, or MON)
73 - Ceph Manager (ceph-mgr, or MGS)
74 - Ceph Metadata Service (ceph-mds, or MDS)
75 - Ceph Object Storage Daemon (ceph-osd, or OSD)
77 TIP: We highly recommend to get familiar with Ceph
78 footnote:[Ceph intro {cephdocs-url}/start/intro/],
80 footnote:[Ceph architecture {cephdocs-url}/architecture/]
82 footnote:[Ceph glossary {cephdocs-url}/glossary].
85 Recommendations for a Healthy Ceph Cluster
86 ------------------------------------------
88 To build a hyper-converged Proxmox + Ceph Cluster, you must use at least three
89 (preferably) identical servers for the setup.
91 Check also the recommendations from
92 {cephdocs-url}/start/hardware-recommendations/[Ceph's website].
94 NOTE: The recommendations below should be seen as a rough guidance for choosing
95 hardware. Therefore, it is still essential to adapt it to your specific needs.
96 You should test your setup and monitor health and performance continuously.
99 Ceph services can be classified into two categories:
100 * Intensive CPU usage, benefiting from high CPU base frequencies and multiple
101 cores. Members of that category are:
102 ** Object Storage Daemon (OSD) services
103 ** Meta Data Service (MDS) used for CephFS
104 * Moderate CPU usage, not needing multiple CPU cores. These are:
105 ** Monitor (MON) services
106 ** Manager (MGR) services
108 As a simple rule of thumb, you should assign at least one CPU core (or thread)
109 to each Ceph service to provide the minimum resources required for stable and
110 durable Ceph performance.
112 For example, if you plan to run a Ceph monitor, a Ceph manager and 6 Ceph OSDs
113 services on a node you should reserve 8 CPU cores purely for Ceph when targeting
114 basic and stable performance.
116 Note that OSDs CPU usage depend mostly from the disks performance. The higher
117 the possible IOPS (**IO** **O**perations per **S**econd) of a disk, the more CPU
118 can be utilized by a OSD service.
119 For modern enterprise SSD disks, like NVMe's that can permanently sustain a high
120 IOPS load over 100'000 with sub millisecond latency, each OSD can use multiple
121 CPU threads, e.g., four to six CPU threads utilized per NVMe backed OSD is
122 likely for very high performance disks.
125 Especially in a hyper-converged setup, the memory consumption needs to be
126 carefully planned out and monitored. In addition to the predicted memory usage
127 of virtual machines and containers, you must also account for having enough
128 memory available for Ceph to provide excellent and stable performance.
130 As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
131 by an OSD. While the usage might be less under normal conditions, it will use
132 most during critical operations like recovery, re-balancing or backfilling.
133 That means that you should avoid maxing out your available memory already on
134 normal operation, but rather leave some headroom to cope with outages.
136 The OSD service itself will use additional memory. The Ceph BlueStore backend of
137 the daemon requires by default **3-5 GiB of memory** (adjustable).
140 We recommend a network bandwidth of at least 10 Gbps, or more, to be used
141 exclusively for Ceph traffic. A meshed network setup
142 footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
143 is also an option for three to five node clusters, if there are no 10+ Gbps
147 The volume of traffic, especially during recovery, will interfere
148 with other services on the same network, especially the latency sensitive {pve}
149 corosync cluster stack can be affected, resulting in possible loss of cluster
150 quorum. Moving the Ceph traffic to dedicated and physical separated networks
151 will avoid such interference, not only for corosync, but also for the networking
152 services provided by any virtual guests.
154 For estimating your bandwidth needs, you need to take the performance of your
155 disks into account.. While a single HDD might not saturate a 1 Gb link, multiple
156 HDD OSDs per node can already saturate 10 Gbps too.
157 If modern NVMe-attached SSDs are used, a single one can already saturate 10 Gbps
158 of bandwidth, or more. For such high-performance setups we recommend at least
159 a 25 Gpbs, while even 40 Gbps or 100+ Gbps might be required to utilize the full
160 performance potential of the underlying disks.
162 If unsure, we recommend using three (physical) separate networks for
163 high-performance setups:
164 * one very high bandwidth (25+ Gbps) network for Ceph (internal) cluster
166 * one high bandwidth (10+ Gpbs) network for Ceph (public) traffic between the
167 ceph server and ceph client storage traffic. Depending on your needs this can
168 also be used to host the virtual guest traffic and the VM live-migration
170 * one medium bandwidth (1 Gbps) exclusive for the latency sensitive corosync
171 cluster communication.
174 When planning the size of your Ceph cluster, it is important to take the
175 recovery time into consideration. Especially with small clusters, recovery
176 might take long. It is recommended that you use SSDs instead of HDDs in small
177 setups to reduce recovery time, minimizing the likelihood of a subsequent
178 failure event during recovery.
180 In general, SSDs will provide more IOPS than spinning disks. With this in mind,
181 in addition to the higher cost, it may make sense to implement a
182 xref:pve_ceph_device_classes[class based] separation of pools. Another way to
183 speed up OSDs is to use a faster disk as a journal or
184 DB/**W**rite-**A**head-**L**og device, see
185 xref:pve_ceph_osds[creating Ceph OSDs].
186 If a faster disk is used for multiple OSDs, a proper balance between OSD
187 and WAL / DB (or journal) disk must be selected, otherwise the faster disk
188 becomes the bottleneck for all linked OSDs.
190 Aside from the disk type, Ceph performs best with an evenly sized, and an evenly
191 distributed amount of disks per node. For example, 4 x 500 GB disks within each
192 node is better than a mixed setup with a single 1 TB and three 250 GB disk.
194 You also need to balance OSD count and single OSD capacity. More capacity
195 allows you to increase storage density, but it also means that a single OSD
196 failure forces Ceph to recover more data at once.
199 As Ceph handles data object redundancy and multiple parallel writes to disks
200 (OSDs) on its own, using a RAID controller normally doesn’t improve
201 performance or availability. On the contrary, Ceph is designed to handle whole
202 disks on it's own, without any abstraction in between. RAID controllers are not
203 designed for the Ceph workload and may complicate things and sometimes even
204 reduce performance, as their write and caching algorithms may interfere with
207 WARNING: Avoid RAID controllers. Use host bus adapter (HBA) instead.
209 [[pve_ceph_install_wizard]]
210 Initial Ceph Installation & Configuration
211 -----------------------------------------
213 Using the Web-based Wizard
214 ~~~~~~~~~~~~~~~~~~~~~~~~~~
216 [thumbnail="screenshot/gui-node-ceph-install.png"]
218 With {pve} you have the benefit of an easy to use installation wizard
219 for Ceph. Click on one of your cluster nodes and navigate to the Ceph
220 section in the menu tree. If Ceph is not already installed, you will see a
221 prompt offering to do so.
223 The wizard is divided into multiple sections, where each needs to
224 finish successfully, in order to use Ceph.
226 First you need to chose which Ceph version you want to install. Prefer the one
227 from your other nodes, or the newest if this is the first node you install
230 After starting the installation, the wizard will download and install all the
231 required packages from {pve}'s Ceph repository.
232 [thumbnail="screenshot/gui-node-ceph-install-wizard-step0.png"]
234 After finishing the installation step, you will need to create a configuration.
235 This step is only needed once per cluster, as this configuration is distributed
236 automatically to all remaining cluster members through {pve}'s clustered
237 xref:chapter_pmxcfs[configuration file system (pmxcfs)].
239 The configuration step includes the following settings:
241 [[pve_ceph_wizard_networks]]
243 * *Public Network:* This network will be used for public storage communication
244 (e.g., for virtual machines using a Ceph RBD backed disk, or a CephFS mount),
245 and communication between the different Ceph services. This setting is
248 Separating your Ceph traffic from the {pve} cluster communication (corosync),
249 and possible the front-facing (public) networks of your virtual guests, is
250 highly recommended. Otherwise, Ceph's high-bandwidth IO-traffic could cause
251 interference with other low-latency dependent services.
253 [thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
255 * *Cluster Network:* Specify to separate the xref:pve_ceph_osds[OSD] replication
256 and heartbeat traffic as well. This setting is optional.
258 Using a physically separated network is recommended, as it will relieve the
259 Ceph public and the virtual guests network, while also providing a significant
260 Ceph performance improvements.
262 The Ceph cluster network can be configured and moved to another physically
263 separated network at a later time.
265 You have two more options which are considered advanced and therefore should
266 only changed if you know what you are doing.
268 * *Number of replicas*: Defines how often an object is replicated.
269 * *Minimum replicas*: Defines the minimum number of required replicas for I/O to
270 be marked as complete.
272 Additionally, you need to choose your first monitor node. This step is required.
274 That's it. You should now see a success page as the last step, with further
275 instructions on how to proceed. Your system is now ready to start using Ceph.
276 To get started, you will need to create some additional xref:pve_ceph_monitors[monitors],
277 xref:pve_ceph_osds[OSDs] and at least one xref:pve_ceph_pools[pool].
279 The rest of this chapter will guide you through getting the most out of
280 your {pve} based Ceph setup. This includes the aforementioned tips and
281 more, such as xref:pveceph_fs[CephFS], which is a helpful addition to your
285 CLI Installation of Ceph Packages
286 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
288 Alternatively to the the recommended {pve} Ceph installation wizard available
289 in the web-interface, you can use the following CLI command on each node:
296 This sets up an `apt` package repository in
297 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
300 Initial Ceph configuration via CLI
301 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
303 Use the {pve} Ceph installation wizard (recommended) or run the
304 following command on one node:
308 pveceph init --network 10.10.10.0/24
311 This creates an initial configuration at `/etc/pve/ceph.conf` with a
312 dedicated network for Ceph. This file is automatically distributed to
313 all {pve} nodes, using xref:chapter_pmxcfs[pmxcfs]. The command also
314 creates a symbolic link at `/etc/ceph/ceph.conf`, which points to that file.
315 Thus, you can simply run Ceph commands without the need to specify a
319 [[pve_ceph_monitors]]
323 [thumbnail="screenshot/gui-ceph-monitor.png"]
325 The Ceph Monitor (MON)
326 footnote:[Ceph Monitor {cephdocs-url}/start/intro/]
327 maintains a master copy of the cluster map. For high availability, you need at
328 least 3 monitors. One monitor will already be installed if you
329 used the installation wizard. You won't need more than 3 monitors, as long
330 as your cluster is small to medium-sized. Only really large clusters will
331 require more than this.
333 [[pveceph_create_mon]]
337 On each node where you want to place a monitor (three monitors are recommended),
338 create one by using the 'Ceph -> Monitor' tab in the GUI or run:
346 [[pveceph_destroy_mon]]
350 To remove a Ceph Monitor via the GUI, first select a node in the tree view and
351 go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**
354 To remove a Ceph Monitor via the CLI, first connect to the node on which the MON
355 is running. Then execute the following command:
361 NOTE: At least three Monitors are needed for quorum.
368 The Manager daemon runs alongside the monitors. It provides an interface to
369 monitor the cluster. Since the release of Ceph luminous, at least one ceph-mgr
370 footnote:[Ceph Manager {cephdocs-url}/mgr/] daemon is
373 [[pveceph_create_mgr]]
377 Multiple Managers can be installed, but only one Manager is active at any given
385 NOTE: It is recommended to install the Ceph Manager on the monitor nodes. For
386 high availability install more then one manager.
389 [[pveceph_destroy_mgr]]
393 To remove a Ceph Manager via the GUI, first select a node in the tree view and
394 go to the **Ceph -> Monitor** panel. Select the Manager and click the
397 To remove a Ceph Monitor via the CLI, first connect to the node on which the
398 Manager is running. Then execute the following command:
404 NOTE: While a manager is not a hard-dependency, it is crucial for a Ceph cluster,
405 as it handles important features like PG-autoscaling, device health monitoring,
412 [thumbnail="screenshot/gui-ceph-osd-status.png"]
414 Ceph **O**bject **S**torage **D**aemons store objects for Ceph over the
415 network. It is recommended to use one OSD per physical disk.
417 [[pve_ceph_osd_create]]
421 You can create an OSD either via the {pve} web-interface or via the CLI using
422 `pveceph`. For example:
426 pveceph osd create /dev/sd[X]
429 TIP: We recommend a Ceph cluster with at least three nodes and at least 12
430 OSDs, evenly distributed among the nodes.
432 If the disk was in use before (for example, for ZFS or as an OSD) you first need
433 to zap all traces of that usage. To remove the partition table, boot sector and
434 any other OSD leftover, you can use the following command:
438 ceph-volume lvm zap /dev/sd[X] --destroy
441 WARNING: The above command will destroy all data on the disk!
445 Starting with the Ceph Kraken release, a new Ceph OSD storage type was
446 introduced called Bluestore
447 footnote:[Ceph Bluestore https://ceph.com/community/new-luminous-bluestore/].
448 This is the default when creating OSDs since Ceph Luminous.
452 pveceph osd create /dev/sd[X]
455 .Block.db and block.wal
457 If you want to use a separate DB/WAL device for your OSDs, you can specify it
458 through the '-db_dev' and '-wal_dev' options. The WAL is placed with the DB, if
459 not specified separately.
463 pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]
466 You can directly choose the size of those with the '-db_size' and '-wal_size'
467 parameters respectively. If they are not given, the following values (in order)
470 * bluestore_block_{db,wal}_size from Ceph configuration...
471 ** ... database, section 'osd'
472 ** ... database, section 'global'
473 ** ... file, section 'osd'
474 ** ... file, section 'global'
475 * 10% (DB)/1% (WAL) of OSD size
477 NOTE: The DB stores BlueStore’s internal metadata, and the WAL is BlueStore’s
478 internal journal or write-ahead log. It is recommended to use a fast SSD or
479 NVRAM for better performance.
483 Before Ceph Luminous, Filestore was used as the default storage type for Ceph OSDs.
484 Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
485 'pveceph' anymore. If you still want to create filestore OSDs, use
486 'ceph-volume' directly.
490 ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
493 [[pve_ceph_osd_destroy]]
497 To remove an OSD via the GUI, first select a {PVE} node in the tree view and go
498 to the **Ceph -> OSD** panel. Then select the OSD to destroy and click the **OUT**
499 button. Once the OSD status has changed from `in` to `out`, click the **STOP**
500 button. Finally, after the status has changed from `up` to `down`, select
501 **Destroy** from the `More` drop-down menu.
503 To remove an OSD via the CLI run the following commands.
508 systemctl stop ceph-osd@<ID>.service
511 NOTE: The first command instructs Ceph not to include the OSD in the data
512 distribution. The second command stops the OSD service. Until this time, no
515 The following command destroys the OSD. Specify the '-cleanup' option to
516 additionally destroy the partition table.
520 pveceph osd destroy <ID>
523 WARNING: The above command will destroy all data on the disk!
530 [thumbnail="screenshot/gui-ceph-pools.png"]
532 A pool is a logical group for storing objects. It holds a collection of objects,
533 known as **P**lacement **G**roups (`PG`, `pg_num`).
536 Create and Edit Pools
537 ~~~~~~~~~~~~~~~~~~~~~
539 You can create and edit pools from the command line or the web-interface of any
540 {pve} host under **Ceph -> Pools**.
542 When no options are given, we set a default of **128 PGs**, a **size of 3
543 replicas** and a **min_size of 2 replicas**, to ensure no data loss occurs if
546 WARNING: **Do not set a min_size of 1**. A replicated pool with min_size of 1
547 allows I/O on an object when it has only 1 replica, which could lead to data
548 loss, incomplete PGs or unfound objects.
550 It is advised that you either enable the PG-Autoscaler or calculate the PG
551 number based on your setup. You can find the formula and the PG calculator
552 footnote:[PG calculator https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/] online. From Ceph Nautilus
553 onward, you can change the number of PGs
554 footnoteref:[placement_groups,Placement Groups
555 {cephdocs-url}/rados/operations/placement-groups/] after the setup.
557 The PG autoscaler footnoteref:[autoscaler,Automated Scaling
558 {cephdocs-url}/rados/operations/placement-groups/#automated-scaling] can
559 automatically scale the PG count for a pool in the background. Setting the
560 `Target Size` or `Target Ratio` advanced parameters helps the PG-Autoscaler to
561 make better decisions.
563 .Example for creating a pool over the CLI
566 pveceph pool create <pool-name> --add_storages
569 TIP: If you would also like to automatically define a storage for your
570 pool, keep the `Add as Storage' checkbox checked in the web-interface, or use the
571 command-line option '--add_storages' at pool creation.
576 [thumbnail="screenshot/gui-ceph-pool-create.png"]
578 The following options are available on pool creation, and partially also when
581 Name:: The name of the pool. This must be unique and can't be changed afterwards.
582 Size:: The number of replicas per object. Ceph always tries to have this many
583 copies of an object. Default: `3`.
584 PG Autoscale Mode:: The automatic PG scaling mode footnoteref:[autoscaler] of
585 the pool. If set to `warn`, it produces a warning message when a pool
586 has a non-optimal PG count. Default: `warn`.
587 Add as Storage:: Configure a VM or container storage using the new pool.
588 Default: `true` (only visible on creation).
591 Min. Size:: The minimum number of replicas per object. Ceph will reject I/O on
592 the pool if a PG has less than this many replicas. Default: `2`.
593 Crush Rule:: The rule to use for mapping object placement in the cluster. These
594 rules define how data is placed within the cluster. See
595 xref:pve_ceph_device_classes[Ceph CRUSH & device classes] for information on
597 # of PGs:: The number of placement groups footnoteref:[placement_groups] that
598 the pool should have at the beginning. Default: `128`.
599 Target Ratio:: The ratio of data that is expected in the pool. The PG
600 autoscaler uses the ratio relative to other ratio sets. It takes precedence
601 over the `target size` if both are set.
602 Target Size:: The estimated amount of data expected in the pool. The PG
603 autoscaler uses this size to estimate the optimal PG count.
604 Min. # of PGs:: The minimum number of placement groups. This setting is used to
605 fine-tune the lower bound of the PG count for that pool. The PG autoscaler
606 will not merge PGs below this threshold.
608 Further information on Ceph pool handling can be found in the Ceph pool
609 operation footnote:[Ceph pool operation
610 {cephdocs-url}/rados/operations/pools/]
614 [[pve_ceph_ec_pools]]
618 Erasure coding (EC) is a form of `forward error correction' codes that allows
619 to recover from a certain amount of data loss. Erasure coded pools can offer
620 more usable space compared to replicated pools, but they do that for the price
623 For comparison: in classic, replicated pools, multiple replicas of the data
624 are stored (`size`) while in erasure coded pool, data is split into `k` data
625 chunks with additional `m` coding (checking) chunks. Those coding chunks can be
626 used to recreate data should data chunks be missing.
628 The number of coding chunks, `m`, defines how many OSDs can be lost without
629 losing any data. The total amount of objects stored is `k + m`.
634 Erasure coded (EC) pools can be created with the `pveceph` CLI tooling.
635 Planning an EC pool needs to account for the fact, that they work differently
636 than replicated pools.
638 The default `min_size` of an EC pool depends on the `m` parameter. If `m = 1`,
639 the `min_size` of the EC pool will be `k`. The `min_size` will be `k + 1` if
640 `m > 1`. The Ceph documentation recommends a conservative `min_size` of `k + 2`
641 footnote:[Ceph Erasure Coded Pool Recovery
642 {cephdocs-url}/rados/operations/erasure-code/#erasure-coded-pool-recovery].
644 If there are less than `min_size` OSDs available, any IO to the pool will be
645 blocked until there are enough OSDs available again.
647 NOTE: When planning an erasure coded pool, keep an eye on the `min_size` as it
648 defines how many OSDs need to be available. Otherwise, IO will be blocked.
650 For example, an EC pool with `k = 2` and `m = 1` will have `size = 3`,
651 `min_size = 2` and will stay operational if one OSD fails. If the pool is
652 configured with `k = 2`, `m = 2`, it will have a `size = 4` and `min_size = 3`
653 and stay operational if one OSD is lost.
655 To create a new EC pool, run the following command:
659 pveceph pool create <pool-name> --erasure-coding k=2,m=1
662 Optional parameters are `failure-domain` and `device-class`. If you
663 need to change any EC profile settings used by the pool, you will have to
664 create a new pool with a new profile.
666 This will create a new EC pool plus the needed replicated pool to store the RBD
667 omap and other metadata. In the end, there will be a `<pool name>-data` and
668 `<pool name>-metada` pool. The default behavior is to create a matching storage
669 configuration as well. If that behavior is not wanted, you can disable it by
670 providing the `--add_storages 0` parameter. When configuring the storage
671 configuration manually, keep in mind that the `data-pool` parameter needs to be
672 set. Only then will the EC pool be used to store the data objects. For example:
674 NOTE: The optional parameters `--size`, `--min_size` and `--crush_rule` will be
675 used for the replicated metadata pool, but not for the erasure coded data pool.
676 If you need to change the `min_size` on the data pool, you can do it later.
677 The `size` and `crush_rule` parameters cannot be changed on erasure coded
680 If there is a need to further customize the EC profile, you can do so by
681 creating it with the Ceph tools directly footnote:[Ceph Erasure Code Profile
682 {cephdocs-url}/rados/operations/erasure-code/#erasure-code-profiles], and
683 specify the profile to use with the `profile` parameter.
688 pveceph pool create <pool-name> --erasure-coding profile=<profile-name>
691 Adding EC Pools as Storage
692 ^^^^^^^^^^^^^^^^^^^^^^^^^^
694 You can add an already existing EC pool as storage to {pve}. It works the same
695 way as adding an `RBD` pool but requires the extra `data-pool` option.
699 pvesm add rbd <storage-name> --pool <replicated-pool> --data-pool <ec-pool>
702 TIP: Do not forget to add the `keyring` and `monhost` option for any external
703 Ceph clusters, not managed by the local {pve} cluster.
708 To destroy a pool via the GUI, select a node in the tree view and go to the
709 **Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**
710 button. To confirm the destruction of the pool, you need to enter the pool name.
712 Run the following command to destroy a pool. Specify the '-remove_storages' to
713 also remove the associated storage.
717 pveceph pool destroy <name>
720 NOTE: Pool deletion runs in the background and can take some time.
721 You will notice the data usage in the cluster decreasing throughout this
728 The PG autoscaler allows the cluster to consider the amount of (expected) data
729 stored in each pool and to choose the appropriate pg_num values automatically.
730 It is available since Ceph Nautilus.
732 You may need to activate the PG autoscaler module before adjustments can take
737 ceph mgr module enable pg_autoscaler
740 The autoscaler is configured on a per pool basis and has the following modes:
743 warn:: A health warning is issued if the suggested `pg_num` value differs too
744 much from the current value.
745 on:: The `pg_num` is adjusted automatically with no need for any manual
747 off:: No automatic `pg_num` adjustments are made, and no warning will be issued
748 if the PG count is not optimal.
750 The scaling factor can be adjusted to facilitate future data storage with the
751 `target_size`, `target_size_ratio` and the `pg_num_min` options.
753 WARNING: By default, the autoscaler considers tuning the PG count of a pool if
754 it is off by a factor of 3. This will lead to a considerable shift in data
755 placement and might introduce a high load on the cluster.
757 You can find a more in-depth introduction to the PG autoscaler on Ceph's Blog -
758 https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/[New in
759 Nautilus: PG merging and autotuning].
762 [[pve_ceph_device_classes]]
763 Ceph CRUSH & device classes
764 ---------------------------
766 [thumbnail="screenshot/gui-ceph-config.png"]
769 https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf] (**C**ontrolled
770 **R**eplication **U**nder **S**calable **H**ashing) algorithm is at the
773 CRUSH calculates where to store and retrieve data from. This has the
774 advantage that no central indexing service is needed. CRUSH works using a map of
775 OSDs, buckets (device locations) and rulesets (data replication) for pools.
777 NOTE: Further information can be found in the Ceph documentation, under the
778 section CRUSH map footnote:[CRUSH map {cephdocs-url}/rados/operations/crush-map/].
780 This map can be altered to reflect different replication hierarchies. The object
781 replicas can be separated (e.g., failure domains), while maintaining the desired
784 A common configuration is to use different classes of disks for different Ceph
785 pools. For this reason, Ceph introduced device classes with luminous, to
786 accommodate the need for easy ruleset generation.
788 The device classes can be seen in the 'ceph osd tree' output. These classes
789 represent their own root bucket, which can be seen with the below command.
793 ceph osd crush tree --show-shadow
796 Example output form the above command:
800 ID CLASS WEIGHT TYPE NAME
801 -16 nvme 2.18307 root default~nvme
802 -13 nvme 0.72769 host sumi1~nvme
803 12 nvme 0.72769 osd.12
804 -14 nvme 0.72769 host sumi2~nvme
805 13 nvme 0.72769 osd.13
806 -15 nvme 0.72769 host sumi3~nvme
807 14 nvme 0.72769 osd.14
808 -1 7.70544 root default
809 -3 2.56848 host sumi1
810 12 nvme 0.72769 osd.12
811 -5 2.56848 host sumi2
812 13 nvme 0.72769 osd.13
813 -7 2.56848 host sumi3
814 14 nvme 0.72769 osd.14
817 To instruct a pool to only distribute objects on a specific device class, you
818 first need to create a ruleset for the device class:
822 ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> <class>
825 [frame="none",grid="none", align="left", cols="30%,70%"]
827 |<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
828 |<root>|which crush root it should belong to (default Ceph root "default")
829 |<failure-domain>|at which failure-domain the objects should be distributed (usually host)
830 |<class>|what type of OSD backing store to use (e.g., nvme, ssd, hdd)
833 Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
837 ceph osd pool set <pool-name> crush_rule <rule-name>
840 TIP: If the pool already contains objects, these must be moved accordingly.
841 Depending on your setup, this may introduce a big performance impact on your
842 cluster. As an alternative, you can create a new pool and move disks separately.
848 [thumbnail="screenshot/gui-ceph-log.png"]
850 Following the setup from the previous sections, you can configure {pve} to use
851 such pools to store VM and Container images. Simply use the GUI to add a new
852 `RBD` storage (see section
853 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
855 You also need to copy the keyring to a predefined location for an external Ceph
856 cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
859 NOTE: The filename needs to be `<storage_id> + `.keyring`, where `<storage_id>` is
860 the expression after 'rbd:' in `/etc/pve/storage.cfg`. In the following example,
861 `my-ceph-storage` is the `<storage_id>`:
865 mkdir /etc/pve/priv/ceph
866 cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
873 Ceph also provides a filesystem, which runs on top of the same object storage as
874 RADOS block devices do. A **M**eta**d**ata **S**erver (`MDS`) is used to map the
875 RADOS backed objects to files and directories, allowing Ceph to provide a
876 POSIX-compliant, replicated filesystem. This allows you to easily configure a
877 clustered, highly available, shared filesystem. Ceph's Metadata Servers
878 guarantee that files are evenly distributed over the entire Ceph cluster. As a
879 result, even cases of high load will not overwhelm a single host, which can be
880 an issue with traditional shared filesystem approaches, for example `NFS`.
882 [thumbnail="screenshot/gui-node-ceph-cephfs-panel.png"]
884 {pve} supports both creating a hyper-converged CephFS and using an existing
885 xref:storage_cephfs[CephFS as storage] to save backups, ISO files, and container
890 Metadata Server (MDS)
891 ~~~~~~~~~~~~~~~~~~~~~
893 CephFS needs at least one Metadata Server to be configured and running, in order
894 to function. You can create an MDS through the {pve} web GUI's `Node
895 -> CephFS` panel or from the command line with:
901 Multiple metadata servers can be created in a cluster, but with the default
902 settings, only one can be active at a time. If an MDS or its node becomes
903 unresponsive (or crashes), another `standby` MDS will get promoted to `active`.
904 You can speed up the handover between the active and standby MDS by using
905 the 'hotstandby' parameter option on creation, or if you have already created it
909 mds standby replay = true
912 in the respective MDS section of `/etc/pve/ceph.conf`. With this enabled, the
913 specified MDS will remain in a `warm` state, polling the active one, so that it
914 can take over faster in case of any issues.
916 NOTE: This active polling will have an additional performance impact on your
917 system and the active `MDS`.
921 Since Luminous (12.2.x) you can have multiple active metadata servers
922 running at once, but this is normally only useful if you have a high amount of
923 clients running in parallel. Otherwise the `MDS` is rarely the bottleneck in a
924 system. If you want to set this up, please refer to the Ceph documentation.
925 footnote:[Configuring multiple active MDS daemons
926 {cephdocs-url}/cephfs/multimds/]
928 [[pveceph_fs_create]]
932 With {pve}'s integration of CephFS, you can easily create a CephFS using the
933 web interface, CLI or an external API interface. Some prerequisites are required
936 .Prerequisites for a successful CephFS setup:
937 - xref:pve_ceph_install[Install Ceph packages] - if this was already done some
938 time ago, you may want to rerun it on an up-to-date system to
939 ensure that all CephFS related packages get installed.
940 - xref:pve_ceph_monitors[Setup Monitors]
941 - xref:pve_ceph_monitors[Setup your OSDs]
942 - xref:pveceph_fs_mds[Setup at least one MDS]
944 After this is complete, you can simply create a CephFS through
945 either the Web GUI's `Node -> CephFS` panel or the command-line tool `pveceph`,
949 pveceph fs create --pg_num 128 --add-storage
952 This creates a CephFS named 'cephfs', using a pool for its data named
953 'cephfs_data' with '128' placement groups and a pool for its metadata named
954 'cephfs_metadata' with one quarter of the data pool's placement groups (`32`).
955 Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
956 Ceph documentation for more information regarding an appropriate placement group
957 number (`pg_num`) for your setup footnoteref:[placement_groups].
958 Additionally, the '--add-storage' parameter will add the CephFS to the {pve}
959 storage configuration after it has been created successfully.
964 WARNING: Destroying a CephFS will render all of its data unusable. This cannot be
967 To completely and gracefully remove a CephFS, the following steps are
970 * Disconnect every non-{PVE} client (e.g. unmount the CephFS in guests).
971 * Disable all related CephFS {PVE} storage entries (to prevent it from being
972 automatically mounted).
973 * Remove all used resources from guests (e.g. ISOs) that are on the CephFS you
975 * Unmount the CephFS storages on all cluster nodes manually with
978 umount /mnt/pve/<STORAGE-NAME>
981 Where `<STORAGE-NAME>` is the name of the CephFS storage in your {PVE}.
983 * Now make sure that no metadata server (`MDS`) is running for that CephFS,
984 either by stopping or destroying them. This can be done through the web
985 interface or via the command-line interface, for the latter you would issue
986 the following command:
989 pveceph stop --service mds.NAME
995 pveceph mds destroy NAME
1000 Note that standby servers will automatically be promoted to active when an
1001 active `MDS` is stopped or removed, so it is best to first stop all standby
1004 * Now you can destroy the CephFS with
1007 pveceph fs destroy NAME --remove-storages --remove-pools
1010 This will automatically destroy the underlying Ceph pools as well as remove
1011 the storages from pve config.
1013 After these steps, the CephFS should be completely removed and if you have
1014 other CephFS instances, the stopped metadata servers can be started again
1023 One of the most common maintenance tasks in Ceph is to replace the disk of an
1024 OSD. If a disk is already in a failed state, then you can go ahead and run
1025 through the steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate
1026 those copies on the remaining OSDs if possible. This rebalancing will start as
1027 soon as an OSD failure is detected or an OSD was actively stopped.
1029 NOTE: With the default size/min_size (3/2) of a pool, recovery only starts when
1030 `size + 1` nodes are available. The reason for this is that the Ceph object
1031 balancer xref:pve_ceph_device_classes[CRUSH] defaults to a full node as
1034 To replace a functioning disk from the GUI, go through the steps in
1035 xref:pve_ceph_osd_destroy[Destroy OSDs]. The only addition is to wait until
1036 the cluster shows 'HEALTH_OK' before stopping the OSD to destroy it.
1038 On the command line, use the following commands:
1041 ceph osd out osd.<id>
1044 You can check with the command below if the OSD can be safely removed.
1047 ceph osd safe-to-destroy osd.<id>
1050 Once the above check tells you that it is safe to remove the OSD, you can
1051 continue with the following commands:
1054 systemctl stop ceph-osd@<id>.service
1055 pveceph osd destroy <id>
1058 Replace the old disk with the new one and use the same procedure as described
1059 in xref:pve_ceph_osd_create[Create OSDs].
1064 It is good practice to run 'fstrim' (discard) regularly on VMs and containers.
1065 This releases data blocks that the filesystem isn’t using anymore. It reduces
1066 data usage and resource load. Most modern operating systems issue such discard
1067 commands to their disks regularly. You only need to ensure that the Virtual
1068 Machines enable the xref:qm_hard_disk_discard[disk discard option].
1074 Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every
1075 object in a PG for its health. There are two forms of Scrubbing, daily
1076 cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
1077 the objects and uses checksums to ensure data integrity. If a running scrub
1078 interferes with business (performance) needs, you can adjust the time when
1079 scrubs footnote:[Ceph scrubbing {cephdocs-url}/rados/configuration/osd-config-ref/#scrubbing]
1083 Ceph Monitoring and Troubleshooting
1084 -----------------------------------
1086 It is important to continuously monitor the health of a Ceph deployment from the
1087 beginning, either by using the Ceph tools or by accessing
1088 the status through the {pve} link:api-viewer/index.html[API].
1090 The following Ceph commands can be used to see if the cluster is healthy
1091 ('HEALTH_OK'), if there are warnings ('HEALTH_WARN'), or even errors
1092 ('HEALTH_ERR'). If the cluster is in an unhealthy state, the status commands
1093 below will also give you an overview of the current events and actions to take.
1096 # single time output
1098 # continuously output status changes (press CTRL+C to stop)
1102 To get a more detailed view, every Ceph service has a log file under
1103 `/var/log/ceph/`. If more detail is required, the log level can be
1104 adjusted footnote:[Ceph log and debugging {cephdocs-url}/rados/troubleshooting/log-and-debug/].
1106 You can find more information about troubleshooting
1107 footnote:[Ceph troubleshooting {cephdocs-url}/rados/troubleshooting/]
1108 a Ceph cluster on the official website.
1112 include::pve-copyright.adoc[]