:pve-toplevel:
endif::manvolnum[]
-[thumbnail="screenshot/gui-ceph-status.png"]
+[thumbnail="screenshot/gui-ceph-status-dashboard.png"]
{pve} unifies your compute and storage systems, that is, you can use the same
physical nodes within a cluster for both computing (processing VMs and
available for Ceph to provide excellent and stable performance.
As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
-by an OSD. Especially during recovery, rebalancing or backfilling.
+by an OSD. Especially during recovery, re-balancing or backfilling.
The daemon itself will use additional memory. The Bluestore backend of the
daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the
setups to reduce recovery time, minimizing the likelihood of a subsequent
failure event during recovery.
-In general SSDs will provide more IOPs than spinning disks. With this in mind,
+In general, SSDs will provide more IOPS than spinning disks. With this in mind,
in addition to the higher cost, it may make sense to implement a
xref:pve_ceph_device_classes[class based] separation of pools. Another way to
speed up OSDs is to use a faster disk as a journal or
-DB/**W**rite-**A**head-**L**og device, see xref:pve_ceph_osds[creating Ceph
-OSDs]. If a faster disk is used for multiple OSDs, a proper balance between OSD
+DB/**W**rite-**A**head-**L**og device, see
+xref:pve_ceph_osds[creating Ceph OSDs].
+If a faster disk is used for multiple OSDs, a proper balance between OSD
and WAL / DB (or journal) disk must be selected, otherwise the faster disk
becomes the bottleneck for all linked OSDs.
Initial Ceph Installation & Configuration
-----------------------------------------
+Using the Web-based Wizard
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
[thumbnail="screenshot/gui-node-ceph-install.png"]
With {pve} you have the benefit of an easy to use installation wizard
prompt offering to do so.
The wizard is divided into multiple sections, where each needs to
-finish successfully, in order to use Ceph. After starting the installation,
-the wizard will download and install all the required packages from {pve}'s Ceph
-repository.
+finish successfully, in order to use Ceph.
+
+First you need to chose which Ceph version you want to install. Prefer the one
+from your other nodes, or the newest if this is the first node you install
+Ceph.
-After finishing the first step, you will need to create a configuration.
+After starting the installation, the wizard will download and install all the
+required packages from {pve}'s Ceph repository.
+[thumbnail="screenshot/gui-node-ceph-install-wizard-step0.png"]
+
+After finishing the installation step, you will need to create a configuration.
This step is only needed once per cluster, as this configuration is distributed
automatically to all remaining cluster members through {pve}'s clustered
xref:chapter_pmxcfs[configuration file system (pmxcfs)].
new Ceph cluster.
[[pve_ceph_install]]
-Installation of Ceph Packages
------------------------------
-Use the {pve} Ceph installation wizard (recommended) or run the following
-command on each node:
+CLI Installation of Ceph Packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alternatively to the the recommended {pve} Ceph installation wizard available
+in the web-interface, you can use the following CLI command on each node:
[source,bash]
----
`/etc/apt/sources.list.d/ceph.list` and installs the required software.
-Create initial Ceph configuration
----------------------------------
-
-[thumbnail="screenshot/gui-ceph-config.png"]
+Initial Ceph configuration via CLI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the {pve} Ceph installation wizard (recommended) or run the
following command on one node:
[[pve_ceph_monitors]]
Ceph Monitor
-----------
+
+[thumbnail="screenshot/gui-ceph-monitor.png"]
+
The Ceph Monitor (MON)
footnote:[Ceph Monitor {cephdocs-url}/start/intro/]
maintains a master copy of the cluster map. For high availability, you need at
as your cluster is small to medium-sized. Only really large clusters will
require more than this.
-
[[pveceph_create_mon]]
Create Monitors
~~~~~~~~~~~~~~~
-[thumbnail="screenshot/gui-ceph-monitor.png"]
-
On each node where you want to place a monitor (three monitors are recommended),
create one by using the 'Ceph -> Monitor' tab in the GUI or run:
[[pve_ceph_osds]]
Ceph OSDs
---------
+
+[thumbnail="screenshot/gui-ceph-osd-status.png"]
+
Ceph **O**bject **S**torage **D**aemons store objects for Ceph over the
network. It is recommended to use one OSD per physical disk.
-NOTE: By default an object is 4 MiB in size.
-
[[pve_ceph_osd_create]]
Create OSDs
~~~~~~~~~~~
-[thumbnail="screenshot/gui-ceph-osd-status.png"]
-
You can create an OSD either via the {pve} web-interface or via the CLI using
`pveceph`. For example:
internal journal or write-ahead log. It is recommended to use a fast SSD or
NVRAM for better performance.
-
.Ceph Filestore
Before Ceph Luminous, Filestore was used as the default storage type for Ceph OSDs.
[[pve_ceph_pools]]
Ceph Pools
----------
+
+[thumbnail="screenshot/gui-ceph-pools.png"]
+
A pool is a logical group for storing objects. It holds a collection of objects,
known as **P**lacement **G**roups (`PG`, `pg_num`).
Create and Edit Pools
~~~~~~~~~~~~~~~~~~~~~
-You can create pools from the command line or the web-interface of any {pve}
-host under **Ceph -> Pools**.
-
-[thumbnail="screenshot/gui-ceph-pools.png"]
+You can create and edit pools from the command line or the web-interface of any
+{pve} host under **Ceph -> Pools**.
When no options are given, we set a default of **128 PGs**, a **size of 3
replicas** and a **min_size of 2 replicas**, to ensure no data loss occurs if
allows I/O on an object when it has only 1 replica, which could lead to data
loss, incomplete PGs or unfound objects.
-It is advised that you calculate the PG number based on your setup. You can
-find the formula and the PG calculator footnote:[PG calculator
-https://ceph.com/pgcalc/] online. From Ceph Nautilus onward, you can change the
-number of PGs footnoteref:[placement_groups,Placement Groups
+It is advised that you either enable the PG-Autoscaler or calculate the PG
+number based on your setup. You can find the formula and the PG calculator
+footnote:[PG calculator https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/] online. From Ceph Nautilus
+onward, you can change the number of PGs
+footnoteref:[placement_groups,Placement Groups
{cephdocs-url}/rados/operations/placement-groups/] after the setup.
-In addition to manual adjustment, the PG autoscaler
-footnoteref:[autoscaler,Automated Scaling
+The PG autoscaler footnoteref:[autoscaler,Automated Scaling
{cephdocs-url}/rados/operations/placement-groups/#automated-scaling] can
-automatically scale the PG count for a pool in the background.
+automatically scale the PG count for a pool in the background. Setting the
+`Target Size` or `Target Ratio` advanced parameters helps the PG-Autoscaler to
+make better decisions.
.Example for creating a pool over the CLI
[source,bash]
pool, keep the `Add as Storage' checkbox checked in the web-interface, or use the
command line option '--add_storages' at pool creation.
-.Base Options
+Pool Options
+^^^^^^^^^^^^
+
+[thumbnail="screenshot/gui-ceph-pool-create.png"]
+
+The following options are available on pool creation, and partially also when
+editing a pool.
+
Name:: The name of the pool. This must be unique and can't be changed afterwards.
Size:: The number of replicas per object. Ceph always tries to have this many
copies of an object. Default: `3`.
device-based rules.
# of PGs:: The number of placement groups footnoteref:[placement_groups] that
the pool should have at the beginning. Default: `128`.
-Target Size Ratio:: The ratio of data that is expected in the pool. The PG
+Target Ratio:: The ratio of data that is expected in the pool. The PG
autoscaler uses the ratio relative to other ratio sets. It takes precedence
over the `target size` if both are set.
Target Size:: The estimated amount of data expected in the pool. The PG
The PG autoscaler allows the cluster to consider the amount of (expected) data
stored in each pool and to choose the appropriate pg_num values automatically.
+It is available since Ceph Nautilus.
You may need to activate the PG autoscaler module before adjustments can take
effect.
[[pve_ceph_device_classes]]
Ceph CRUSH & device classes
---------------------------
+
+[thumbnail="screenshot/gui-ceph-config.png"]
+
The footnote:[CRUSH
https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf] (**C**ontrolled
**R**eplication **U**nder **S**calable **H**ashing) algorithm is at the
section CRUSH map footnote:[CRUSH map {cephdocs-url}/rados/operations/crush-map/].
This map can be altered to reflect different replication hierarchies. The object
-replicas can be separated (eg. failure domains), while maintaining the desired
+replicas can be separated (e.g., failure domains), while maintaining the desired
distribution.
A common configuration is to use different classes of disks for different Ceph
|<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
|<root>|which crush root it should belong to (default ceph root "default")
|<failure-domain>|at which failure-domain the objects should be distributed (usually host)
-|<class>|what type of OSD backing store to use (eg. nvme, ssd, hdd)
+|<class>|what type of OSD backing store to use (e.g., nvme, ssd, hdd)
|===
Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
Following the setup from the previous sections, you can configure {pve} to use
such pools to store VM and Container images. Simply use the GUI to add a new
-`RBD` storage (see section xref:ceph_rados_block_devices[Ceph RADOS Block
-Devices (RBD)]).
+`RBD` storage (see section
+xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
You also need to copy the keyring to a predefined location for an external Ceph
cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
projects / pve-docs.git / blobdiff