10 pveceph - Manage Ceph Services on Proxmox VE Nodes
15 include::pveceph.1-synopsis.adoc[]
21 Deploy Hyper-Converged Ceph Cluster
22 ===================================
26 [thumbnail="screenshot/gui-ceph-status.png"]
28 {pve} unifies your compute and storage systems, that is, you can use the same
29 physical nodes within a cluster for both computing (processing VMs and
30 containers) and replicated storage. The traditional silos of compute and
31 storage resources can be wrapped up into a single hyper-converged appliance.
32 Separate storage networks (SANs) and connections via network attached storage
33 (NAS) disappear. With the integration of Ceph, an open source software-defined
34 storage platform, {pve} has the ability to run and manage Ceph storage directly
35 on the hypervisor nodes.
37 Ceph is a distributed object store and file system designed to provide
38 excellent performance, reliability and scalability.
40 .Some advantages of Ceph on {pve} are:
41 - Easy setup and management via CLI and GUI
45 - Scalable to the exabyte level
46 - Setup pools with different performance and redundancy characteristics
47 - Data is replicated, making it fault tolerant
48 - Runs on commodity hardware
49 - No need for hardware RAID controllers
52 For small to medium-sized deployments, it is possible to install a Ceph server for
53 RADOS Block Devices (RBD) directly on your {pve} cluster nodes (see
54 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]). Recent
55 hardware has a lot of CPU power and RAM, so running storage services
56 and VMs on the same node is possible.
58 To simplify management, we provide 'pveceph' - a tool for installing and
59 managing {ceph} services on {pve} nodes.
61 .Ceph consists of multiple Daemons, for use as an RBD storage:
62 - Ceph Monitor (ceph-mon)
63 - Ceph Manager (ceph-mgr)
64 - Ceph OSD (ceph-osd; Object Storage Daemon)
66 TIP: We highly recommend to get familiar with Ceph
67 footnote:[Ceph intro {cephdocs-url}/start/intro/],
69 footnote:[Ceph architecture {cephdocs-url}/architecture/]
71 footnote:[Ceph glossary {cephdocs-url}/glossary].
77 To build a hyper-converged Proxmox + Ceph Cluster, you must use at least
78 three (preferably) identical servers for the setup.
80 Check also the recommendations from
81 {cephdocs-url}/start/hardware-recommendations/[Ceph's website].
84 A high CPU core frequency reduces latency and should be preferred. As a simple
85 rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
86 provide enough resources for stable and durable Ceph performance.
89 Especially in a hyper-converged setup, the memory consumption needs to be
90 carefully monitored. In addition to the predicted memory usage of virtual
91 machines and containers, you must also account for having enough memory
92 available for Ceph to provide excellent and stable performance.
94 As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
95 by an OSD. Especially during recovery, rebalancing or backfilling.
97 The daemon itself will use additional memory. The Bluestore backend of the
98 daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the
99 legacy Filestore backend uses the OS page cache and the memory consumption is
100 generally related to PGs of an OSD daemon.
103 We recommend a network bandwidth of at least 10 GbE or more, which is used
104 exclusively for Ceph. A meshed network setup
105 footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
106 is also an option if there are no 10 GbE switches available.
108 The volume of traffic, especially during recovery, will interfere with other
109 services on the same network and may even break the {pve} cluster stack.
111 Furthermore, you should estimate your bandwidth needs. While one HDD might not
112 saturate a 1 Gb link, multiple HDD OSDs per node can, and modern NVMe SSDs will
113 even saturate 10 Gbps of bandwidth quickly. Deploying a network capable of even
114 more bandwidth will ensure that this isn't your bottleneck and won't be anytime
115 soon. 25, 40 or even 100 Gbps are possible.
118 When planning the size of your Ceph cluster, it is important to take the
119 recovery time into consideration. Especially with small clusters, recovery
120 might take long. It is recommended that you use SSDs instead of HDDs in small
121 setups to reduce recovery time, minimizing the likelihood of a subsequent
122 failure event during recovery.
124 In general SSDs will provide more IOPs than spinning disks. With this in mind,
125 in addition to the higher cost, it may make sense to implement a
126 xref:pve_ceph_device_classes[class based] separation of pools. Another way to
127 speed up OSDs is to use a faster disk as a journal or
128 DB/**W**rite-**A**head-**L**og device, see
129 xref:pve_ceph_osds[creating Ceph OSDs].
130 If a faster disk is used for multiple OSDs, a proper balance between OSD
131 and WAL / DB (or journal) disk must be selected, otherwise the faster disk
132 becomes the bottleneck for all linked OSDs.
134 Aside from the disk type, Ceph performs best with an even sized and distributed
135 amount of disks per node. For example, 4 x 500 GB disks within each node is
136 better than a mixed setup with a single 1 TB and three 250 GB disk.
138 You also need to balance OSD count and single OSD capacity. More capacity
139 allows you to increase storage density, but it also means that a single OSD
140 failure forces Ceph to recover more data at once.
143 As Ceph handles data object redundancy and multiple parallel writes to disks
144 (OSDs) on its own, using a RAID controller normally doesn’t improve
145 performance or availability. On the contrary, Ceph is designed to handle whole
146 disks on it's own, without any abstraction in between. RAID controllers are not
147 designed for the Ceph workload and may complicate things and sometimes even
148 reduce performance, as their write and caching algorithms may interfere with
151 WARNING: Avoid RAID controllers. Use host bus adapter (HBA) instead.
153 NOTE: The above recommendations should be seen as a rough guidance for choosing
154 hardware. Therefore, it is still essential to adapt it to your specific needs.
155 You should test your setup and monitor health and performance continuously.
157 [[pve_ceph_install_wizard]]
158 Initial Ceph Installation & Configuration
159 -----------------------------------------
161 Using the Web-based Wizard
162 ~~~~~~~~~~~~~~~~~~~~~~~~~~
164 [thumbnail="screenshot/gui-node-ceph-install.png"]
166 With {pve} you have the benefit of an easy to use installation wizard
167 for Ceph. Click on one of your cluster nodes and navigate to the Ceph
168 section in the menu tree. If Ceph is not already installed, you will see a
169 prompt offering to do so.
171 The wizard is divided into multiple sections, where each needs to
172 finish successfully, in order to use Ceph.
174 First you need to chose which Ceph version you want to install. Prefer the one
175 from your other nodes, or the newest if this is the first node you install
178 After starting the installation, the wizard will download and install all the
179 required packages from {pve}'s Ceph repository.
181 After finishing the installation step, you will need to create a configuration.
182 This step is only needed once per cluster, as this configuration is distributed
183 automatically to all remaining cluster members through {pve}'s clustered
184 xref:chapter_pmxcfs[configuration file system (pmxcfs)].
186 The configuration step includes the following settings:
188 * *Public Network:* You can set up a dedicated network for Ceph. This
189 setting is required. Separating your Ceph traffic is highly recommended.
190 Otherwise, it could cause trouble with other latency dependent services,
191 for example, cluster communication may decrease Ceph's performance.
193 [thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
195 * *Cluster Network:* As an optional step, you can go even further and
196 separate the xref:pve_ceph_osds[OSD] replication & heartbeat traffic
197 as well. This will relieve the public network and could lead to
198 significant performance improvements, especially in large clusters.
200 You have two more options which are considered advanced and therefore
201 should only changed if you know what you are doing.
203 * *Number of replicas*: Defines how often an object is replicated
204 * *Minimum replicas*: Defines the minimum number of required replicas
205 for I/O to be marked as complete.
207 Additionally, you need to choose your first monitor node. This step is required.
209 That's it. You should now see a success page as the last step, with further
210 instructions on how to proceed. Your system is now ready to start using Ceph.
211 To get started, you will need to create some additional xref:pve_ceph_monitors[monitors],
212 xref:pve_ceph_osds[OSDs] and at least one xref:pve_ceph_pools[pool].
214 The rest of this chapter will guide you through getting the most out of
215 your {pve} based Ceph setup. This includes the aforementioned tips and
216 more, such as xref:pveceph_fs[CephFS], which is a helpful addition to your
220 CLI Installation of Ceph Packages
221 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
223 Alternatively to the the recommended {pve} Ceph installation wizard available
224 in the web-interface, you can use the following CLI command on each node:
231 This sets up an `apt` package repository in
232 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
235 Initial Ceph configuration via CLI
236 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
238 Use the {pve} Ceph installation wizard (recommended) or run the
239 following command on one node:
243 pveceph init --network 10.10.10.0/24
246 This creates an initial configuration at `/etc/pve/ceph.conf` with a
247 dedicated network for Ceph. This file is automatically distributed to
248 all {pve} nodes, using xref:chapter_pmxcfs[pmxcfs]. The command also
249 creates a symbolic link at `/etc/ceph/ceph.conf`, which points to that file.
250 Thus, you can simply run Ceph commands without the need to specify a
254 [[pve_ceph_monitors]]
258 [thumbnail="screenshot/gui-ceph-monitor.png"]
260 The Ceph Monitor (MON)
261 footnote:[Ceph Monitor {cephdocs-url}/start/intro/]
262 maintains a master copy of the cluster map. For high availability, you need at
263 least 3 monitors. One monitor will already be installed if you
264 used the installation wizard. You won't need more than 3 monitors, as long
265 as your cluster is small to medium-sized. Only really large clusters will
266 require more than this.
268 [[pveceph_create_mon]]
272 On each node where you want to place a monitor (three monitors are recommended),
273 create one by using the 'Ceph -> Monitor' tab in the GUI or run:
281 [[pveceph_destroy_mon]]
285 To remove a Ceph Monitor via the GUI, first select a node in the tree view and
286 go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**
289 To remove a Ceph Monitor via the CLI, first connect to the node on which the MON
290 is running. Then execute the following command:
296 NOTE: At least three Monitors are needed for quorum.
303 The Manager daemon runs alongside the monitors. It provides an interface to
304 monitor the cluster. Since the release of Ceph luminous, at least one ceph-mgr
305 footnote:[Ceph Manager {cephdocs-url}/mgr/] daemon is
308 [[pveceph_create_mgr]]
312 Multiple Managers can be installed, but only one Manager is active at any given
320 NOTE: It is recommended to install the Ceph Manager on the monitor nodes. For
321 high availability install more then one manager.
324 [[pveceph_destroy_mgr]]
328 To remove a Ceph Manager via the GUI, first select a node in the tree view and
329 go to the **Ceph -> Monitor** panel. Select the Manager and click the
332 To remove a Ceph Monitor via the CLI, first connect to the node on which the
333 Manager is running. Then execute the following command:
339 NOTE: While a manager is not a hard-dependency, it is crucial for a Ceph cluster,
340 as it handles important features like PG-autoscaling, device health monitoring,
347 [thumbnail="screenshot/gui-ceph-osd-status.png"]
349 Ceph **O**bject **S**torage **D**aemons store objects for Ceph over the
350 network. It is recommended to use one OSD per physical disk.
352 [[pve_ceph_osd_create]]
356 You can create an OSD either via the {pve} web-interface or via the CLI using
357 `pveceph`. For example:
361 pveceph osd create /dev/sd[X]
364 TIP: We recommend a Ceph cluster with at least three nodes and at least 12
365 OSDs, evenly distributed among the nodes.
367 If the disk was in use before (for example, for ZFS or as an OSD) you first need
368 to zap all traces of that usage. To remove the partition table, boot sector and
369 any other OSD leftover, you can use the following command:
373 ceph-volume lvm zap /dev/sd[X] --destroy
376 WARNING: The above command will destroy all data on the disk!
380 Starting with the Ceph Kraken release, a new Ceph OSD storage type was
381 introduced called Bluestore
382 footnote:[Ceph Bluestore https://ceph.com/community/new-luminous-bluestore/].
383 This is the default when creating OSDs since Ceph Luminous.
387 pveceph osd create /dev/sd[X]
390 .Block.db and block.wal
392 If you want to use a separate DB/WAL device for your OSDs, you can specify it
393 through the '-db_dev' and '-wal_dev' options. The WAL is placed with the DB, if
394 not specified separately.
398 pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]
401 You can directly choose the size of those with the '-db_size' and '-wal_size'
402 parameters respectively. If they are not given, the following values (in order)
405 * bluestore_block_{db,wal}_size from Ceph configuration...
406 ** ... database, section 'osd'
407 ** ... database, section 'global'
408 ** ... file, section 'osd'
409 ** ... file, section 'global'
410 * 10% (DB)/1% (WAL) of OSD size
412 NOTE: The DB stores BlueStore’s internal metadata, and the WAL is BlueStore’s
413 internal journal or write-ahead log. It is recommended to use a fast SSD or
414 NVRAM for better performance.
418 Before Ceph Luminous, Filestore was used as the default storage type for Ceph OSDs.
419 Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
420 'pveceph' anymore. If you still want to create filestore OSDs, use
421 'ceph-volume' directly.
425 ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
428 [[pve_ceph_osd_destroy]]
432 To remove an OSD via the GUI, first select a {PVE} node in the tree view and go
433 to the **Ceph -> OSD** panel. Then select the OSD to destroy and click the **OUT**
434 button. Once the OSD status has changed from `in` to `out`, click the **STOP**
435 button. Finally, after the status has changed from `up` to `down`, select
436 **Destroy** from the `More` drop-down menu.
438 To remove an OSD via the CLI run the following commands.
443 systemctl stop ceph-osd@<ID>.service
446 NOTE: The first command instructs Ceph not to include the OSD in the data
447 distribution. The second command stops the OSD service. Until this time, no
450 The following command destroys the OSD. Specify the '-cleanup' option to
451 additionally destroy the partition table.
455 pveceph osd destroy <ID>
458 WARNING: The above command will destroy all data on the disk!
464 A pool is a logical group for storing objects. It holds a collection of objects,
465 known as **P**lacement **G**roups (`PG`, `pg_num`).
468 Create and Edit Pools
469 ~~~~~~~~~~~~~~~~~~~~~
471 You can create and edit pools from the command line or the web-interface of any
472 {pve} host under **Ceph -> Pools**.
474 [thumbnail="screenshot/gui-ceph-pools.png"]
476 When no options are given, we set a default of **128 PGs**, a **size of 3
477 replicas** and a **min_size of 2 replicas**, to ensure no data loss occurs if
480 WARNING: **Do not set a min_size of 1**. A replicated pool with min_size of 1
481 allows I/O on an object when it has only 1 replica, which could lead to data
482 loss, incomplete PGs or unfound objects.
484 It is advised that you either enable the PG-Autoscaler or calculate the PG
485 number based on your setup. You can find the formula and the PG calculator
486 footnote:[PG calculator https://ceph.com/pgcalc/] online. From Ceph Nautilus
487 onward, you can change the number of PGs
488 footnoteref:[placement_groups,Placement Groups
489 {cephdocs-url}/rados/operations/placement-groups/] after the setup.
491 The PG autoscaler footnoteref:[autoscaler,Automated Scaling
492 {cephdocs-url}/rados/operations/placement-groups/#automated-scaling] can
493 automatically scale the PG count for a pool in the background. Setting the
494 `Target Size` or `Target Ratio` advanced parameters helps the PG-Autoscaler to
495 make better decisions.
497 .Example for creating a pool over the CLI
500 pveceph pool create <name> --add_storages
503 TIP: If you would also like to automatically define a storage for your
504 pool, keep the `Add as Storage' checkbox checked in the web-interface, or use the
505 command line option '--add_storages' at pool creation.
510 The following options are available on pool creation, and partially also when
513 Name:: The name of the pool. This must be unique and can't be changed afterwards.
514 Size:: The number of replicas per object. Ceph always tries to have this many
515 copies of an object. Default: `3`.
516 PG Autoscale Mode:: The automatic PG scaling mode footnoteref:[autoscaler] of
517 the pool. If set to `warn`, it produces a warning message when a pool
518 has a non-optimal PG count. Default: `warn`.
519 Add as Storage:: Configure a VM or container storage using the new pool.
520 Default: `true` (only visible on creation).
523 Min. Size:: The minimum number of replicas per object. Ceph will reject I/O on
524 the pool if a PG has less than this many replicas. Default: `2`.
525 Crush Rule:: The rule to use for mapping object placement in the cluster. These
526 rules define how data is placed within the cluster. See
527 xref:pve_ceph_device_classes[Ceph CRUSH & device classes] for information on
529 # of PGs:: The number of placement groups footnoteref:[placement_groups] that
530 the pool should have at the beginning. Default: `128`.
531 Target Ratio:: The ratio of data that is expected in the pool. The PG
532 autoscaler uses the ratio relative to other ratio sets. It takes precedence
533 over the `target size` if both are set.
534 Target Size:: The estimated amount of data expected in the pool. The PG
535 autoscaler uses this size to estimate the optimal PG count.
536 Min. # of PGs:: The minimum number of placement groups. This setting is used to
537 fine-tune the lower bound of the PG count for that pool. The PG autoscaler
538 will not merge PGs below this threshold.
540 Further information on Ceph pool handling can be found in the Ceph pool
541 operation footnote:[Ceph pool operation
542 {cephdocs-url}/rados/operations/pools/]
549 To destroy a pool via the GUI, select a node in the tree view and go to the
550 **Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**
551 button. To confirm the destruction of the pool, you need to enter the pool name.
553 Run the following command to destroy a pool. Specify the '-remove_storages' to
554 also remove the associated storage.
558 pveceph pool destroy <name>
561 NOTE: Pool deletion runs in the background and can take some time.
562 You will notice the data usage in the cluster decreasing throughout this
569 The PG autoscaler allows the cluster to consider the amount of (expected) data
570 stored in each pool and to choose the appropriate pg_num values automatically.
571 It is available since Ceph Nautilus.
573 You may need to activate the PG autoscaler module before adjustments can take
578 ceph mgr module enable pg_autoscaler
581 The autoscaler is configured on a per pool basis and has the following modes:
584 warn:: A health warning is issued if the suggested `pg_num` value differs too
585 much from the current value.
586 on:: The `pg_num` is adjusted automatically with no need for any manual
588 off:: No automatic `pg_num` adjustments are made, and no warning will be issued
589 if the PG count is not optimal.
591 The scaling factor can be adjusted to facilitate future data storage with the
592 `target_size`, `target_size_ratio` and the `pg_num_min` options.
594 WARNING: By default, the autoscaler considers tuning the PG count of a pool if
595 it is off by a factor of 3. This will lead to a considerable shift in data
596 placement and might introduce a high load on the cluster.
598 You can find a more in-depth introduction to the PG autoscaler on Ceph's Blog -
599 https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/[New in
600 Nautilus: PG merging and autotuning].
603 [[pve_ceph_device_classes]]
604 Ceph CRUSH & device classes
605 ---------------------------
607 [thumbnail="screenshot/gui-ceph-config.png"]
610 https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf] (**C**ontrolled
611 **R**eplication **U**nder **S**calable **H**ashing) algorithm is at the
614 CRUSH calculates where to store and retrieve data from. This has the
615 advantage that no central indexing service is needed. CRUSH works using a map of
616 OSDs, buckets (device locations) and rulesets (data replication) for pools.
618 NOTE: Further information can be found in the Ceph documentation, under the
619 section CRUSH map footnote:[CRUSH map {cephdocs-url}/rados/operations/crush-map/].
621 This map can be altered to reflect different replication hierarchies. The object
622 replicas can be separated (eg. failure domains), while maintaining the desired
625 A common configuration is to use different classes of disks for different Ceph
626 pools. For this reason, Ceph introduced device classes with luminous, to
627 accommodate the need for easy ruleset generation.
629 The device classes can be seen in the 'ceph osd tree' output. These classes
630 represent their own root bucket, which can be seen with the below command.
634 ceph osd crush tree --show-shadow
637 Example output form the above command:
641 ID CLASS WEIGHT TYPE NAME
642 -16 nvme 2.18307 root default~nvme
643 -13 nvme 0.72769 host sumi1~nvme
644 12 nvme 0.72769 osd.12
645 -14 nvme 0.72769 host sumi2~nvme
646 13 nvme 0.72769 osd.13
647 -15 nvme 0.72769 host sumi3~nvme
648 14 nvme 0.72769 osd.14
649 -1 7.70544 root default
650 -3 2.56848 host sumi1
651 12 nvme 0.72769 osd.12
652 -5 2.56848 host sumi2
653 13 nvme 0.72769 osd.13
654 -7 2.56848 host sumi3
655 14 nvme 0.72769 osd.14
658 To instruct a pool to only distribute objects on a specific device class, you
659 first need to create a ruleset for the device class:
663 ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> <class>
666 [frame="none",grid="none", align="left", cols="30%,70%"]
668 |<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
669 |<root>|which crush root it should belong to (default ceph root "default")
670 |<failure-domain>|at which failure-domain the objects should be distributed (usually host)
671 |<class>|what type of OSD backing store to use (eg. nvme, ssd, hdd)
674 Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
678 ceph osd pool set <pool-name> crush_rule <rule-name>
681 TIP: If the pool already contains objects, these must be moved accordingly.
682 Depending on your setup, this may introduce a big performance impact on your
683 cluster. As an alternative, you can create a new pool and move disks separately.
689 [thumbnail="screenshot/gui-ceph-log.png"]
691 Following the setup from the previous sections, you can configure {pve} to use
692 such pools to store VM and Container images. Simply use the GUI to add a new
693 `RBD` storage (see section
694 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
696 You also need to copy the keyring to a predefined location for an external Ceph
697 cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
700 NOTE: The filename needs to be `<storage_id> + `.keyring`, where `<storage_id>` is
701 the expression after 'rbd:' in `/etc/pve/storage.cfg`. In the following example,
702 `my-ceph-storage` is the `<storage_id>`:
706 mkdir /etc/pve/priv/ceph
707 cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
714 Ceph also provides a filesystem, which runs on top of the same object storage as
715 RADOS block devices do. A **M**eta**d**ata **S**erver (`MDS`) is used to map the
716 RADOS backed objects to files and directories, allowing Ceph to provide a
717 POSIX-compliant, replicated filesystem. This allows you to easily configure a
718 clustered, highly available, shared filesystem. Ceph's Metadata Servers
719 guarantee that files are evenly distributed over the entire Ceph cluster. As a
720 result, even cases of high load will not overwhelm a single host, which can be
721 an issue with traditional shared filesystem approaches, for example `NFS`.
723 [thumbnail="screenshot/gui-node-ceph-cephfs-panel.png"]
725 {pve} supports both creating a hyper-converged CephFS and using an existing
726 xref:storage_cephfs[CephFS as storage] to save backups, ISO files, and container
731 Metadata Server (MDS)
732 ~~~~~~~~~~~~~~~~~~~~~
734 CephFS needs at least one Metadata Server to be configured and running, in order
735 to function. You can create an MDS through the {pve} web GUI's `Node
736 -> CephFS` panel or from the command line with:
742 Multiple metadata servers can be created in a cluster, but with the default
743 settings, only one can be active at a time. If an MDS or its node becomes
744 unresponsive (or crashes), another `standby` MDS will get promoted to `active`.
745 You can speed up the handover between the active and standby MDS by using
746 the 'hotstandby' parameter option on creation, or if you have already created it
750 mds standby replay = true
753 in the respective MDS section of `/etc/pve/ceph.conf`. With this enabled, the
754 specified MDS will remain in a `warm` state, polling the active one, so that it
755 can take over faster in case of any issues.
757 NOTE: This active polling will have an additional performance impact on your
758 system and the active `MDS`.
762 Since Luminous (12.2.x) you can have multiple active metadata servers
763 running at once, but this is normally only useful if you have a high amount of
764 clients running in parallel. Otherwise the `MDS` is rarely the bottleneck in a
765 system. If you want to set this up, please refer to the Ceph documentation.
766 footnote:[Configuring multiple active MDS daemons
767 {cephdocs-url}/cephfs/multimds/]
769 [[pveceph_fs_create]]
773 With {pve}'s integration of CephFS, you can easily create a CephFS using the
774 web interface, CLI or an external API interface. Some prerequisites are required
777 .Prerequisites for a successful CephFS setup:
778 - xref:pve_ceph_install[Install Ceph packages] - if this was already done some
779 time ago, you may want to rerun it on an up-to-date system to
780 ensure that all CephFS related packages get installed.
781 - xref:pve_ceph_monitors[Setup Monitors]
782 - xref:pve_ceph_monitors[Setup your OSDs]
783 - xref:pveceph_fs_mds[Setup at least one MDS]
785 After this is complete, you can simply create a CephFS through
786 either the Web GUI's `Node -> CephFS` panel or the command line tool `pveceph`,
790 pveceph fs create --pg_num 128 --add-storage
793 This creates a CephFS named 'cephfs', using a pool for its data named
794 'cephfs_data' with '128' placement groups and a pool for its metadata named
795 'cephfs_metadata' with one quarter of the data pool's placement groups (`32`).
796 Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
797 Ceph documentation for more information regarding an appropriate placement group
798 number (`pg_num`) for your setup footnoteref:[placement_groups].
799 Additionally, the '--add-storage' parameter will add the CephFS to the {pve}
800 storage configuration after it has been created successfully.
805 WARNING: Destroying a CephFS will render all of its data unusable. This cannot be
808 If you really want to destroy an existing CephFS, you first need to stop or
809 destroy all metadata servers (`M̀DS`). You can destroy them either via the web
810 interface or via the command line interface, by issuing
813 pveceph mds destroy NAME
815 on each {pve} node hosting an MDS daemon.
817 Then, you can remove (destroy) the CephFS by issuing
820 ceph fs rm NAME --yes-i-really-mean-it
822 on a single node hosting Ceph. After this, you may want to remove the created
823 data and metadata pools, this can be done either over the Web GUI or the CLI
827 pveceph pool destroy NAME
837 One of the most common maintenance tasks in Ceph is to replace the disk of an
838 OSD. If a disk is already in a failed state, then you can go ahead and run
839 through the steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate
840 those copies on the remaining OSDs if possible. This rebalancing will start as
841 soon as an OSD failure is detected or an OSD was actively stopped.
843 NOTE: With the default size/min_size (3/2) of a pool, recovery only starts when
844 `size + 1` nodes are available. The reason for this is that the Ceph object
845 balancer xref:pve_ceph_device_classes[CRUSH] defaults to a full node as
848 To replace a functioning disk from the GUI, go through the steps in
849 xref:pve_ceph_osd_destroy[Destroy OSDs]. The only addition is to wait until
850 the cluster shows 'HEALTH_OK' before stopping the OSD to destroy it.
852 On the command line, use the following commands:
855 ceph osd out osd.<id>
858 You can check with the command below if the OSD can be safely removed.
861 ceph osd safe-to-destroy osd.<id>
864 Once the above check tells you that it is safe to remove the OSD, you can
865 continue with the following commands:
868 systemctl stop ceph-osd@<id>.service
869 pveceph osd destroy <id>
872 Replace the old disk with the new one and use the same procedure as described
873 in xref:pve_ceph_osd_create[Create OSDs].
878 It is good practice to run 'fstrim' (discard) regularly on VMs and containers.
879 This releases data blocks that the filesystem isn’t using anymore. It reduces
880 data usage and resource load. Most modern operating systems issue such discard
881 commands to their disks regularly. You only need to ensure that the Virtual
882 Machines enable the xref:qm_hard_disk_discard[disk discard option].
888 Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every
889 object in a PG for its health. There are two forms of Scrubbing, daily
890 cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
891 the objects and uses checksums to ensure data integrity. If a running scrub
892 interferes with business (performance) needs, you can adjust the time when
893 scrubs footnote:[Ceph scrubbing {cephdocs-url}/rados/configuration/osd-config-ref/#scrubbing]
897 Ceph Monitoring and Troubleshooting
898 -----------------------------------
900 It is important to continuously monitor the health of a Ceph deployment from the
901 beginning, either by using the Ceph tools or by accessing
902 the status through the {pve} link:api-viewer/index.html[API].
904 The following Ceph commands can be used to see if the cluster is healthy
905 ('HEALTH_OK'), if there are warnings ('HEALTH_WARN'), or even errors
906 ('HEALTH_ERR'). If the cluster is in an unhealthy state, the status commands
907 below will also give you an overview of the current events and actions to take.
912 # continuously output status changes (press CTRL+C to stop)
916 To get a more detailed view, every Ceph service has a log file under
917 `/var/log/ceph/`. If more detail is required, the log level can be
918 adjusted footnote:[Ceph log and debugging {cephdocs-url}/rados/troubleshooting/log-and-debug/].
920 You can find more information about troubleshooting
921 footnote:[Ceph troubleshooting {cephdocs-url}/rados/troubleshooting/]
922 a Ceph cluster on the official website.
926 include::pve-copyright.adoc[]