To simplify management, we provide 'pveceph' - a tool to install and
manage {ceph} services on {pve} nodes.
-.Ceph consists of a couple of Daemons footnote:[Ceph intro https://docs.ceph.com/docs/{ceph_codename}/start/intro/], for use as a RBD storage:
+.Ceph consists of a couple of Daemons, for use as a RBD storage:
- Ceph Monitor (ceph-mon)
- Ceph Manager (ceph-mgr)
- Ceph OSD (ceph-osd; Object Storage Daemon)
-TIP: We highly recommend to get familiar with Ceph's architecture
-footnote:[Ceph architecture https://docs.ceph.com/docs/{ceph_codename}/architecture/]
+TIP: We highly recommend to get familiar with Ceph
+footnote:[Ceph intro {cephdocs-url}/start/intro/],
+its architecture
+footnote:[Ceph architecture {cephdocs-url}/architecture/]
and vocabulary
-footnote:[Ceph glossary https://docs.ceph.com/docs/{ceph_codename}/glossary].
+footnote:[Ceph glossary {cephdocs-url}/glossary].
Precondition
three (preferably) identical servers for the setup.
Check also the recommendations from
-https://docs.ceph.com/docs/{ceph_codename}/start/hardware-recommendations/[Ceph's website].
+{cephdocs-url}/start/hardware-recommendations/[Ceph's website].
.CPU
Higher CPU core frequency reduce latency and should be preferred. As a simple
Ceph Monitor
-----------
The Ceph Monitor (MON)
-footnote:[Ceph Monitor https://docs.ceph.com/docs/{ceph_codename}/start/intro/]
+footnote:[Ceph Monitor {cephdocs-url}/start/intro/]
maintains a master copy of the cluster map. For high availability you need to
have at least 3 monitors. One monitor will already be installed if you
used the installation wizard. You won't need more than 3 monitors as long
------------
The Manager daemon runs alongside the monitors. It provides an interface to
monitor the cluster. Since the Ceph luminous release at least one ceph-mgr
-footnote:[Ceph Manager https://docs.ceph.com/docs/{ceph_codename}/mgr/] daemon is
+footnote:[Ceph Manager {cephdocs-url}/mgr/] daemon is
required.
[[pveceph_create_mgr]]
NOTE: The default number of PGs works for 2-5 disks. Ceph throws a
'HEALTH_WARNING' if you have too few or too many PGs in your cluster.
-It is advised to calculate the PG number depending on your setup, you can find
-the formula and the PG calculator footnote:[PG calculator
-https://ceph.com/pgcalc/] online. From Ceph Nautilus onwards it is possible to
-increase and decrease the number of PGs later on footnote:[Placement Groups
-https://docs.ceph.com/docs/{ceph_codename}/rados/operations/placement-groups/].
+WARNING: **Do not set a min_size of 1**. A replicated pool with min_size of 1
+allows I/O on an object when it has only 1 replica which could lead to data
+loss, incomplete PGs or unfound objects.
+It is advised that you calculate the PG number based on your setup. You can
+find the formula and the PG calculator footnote:[PG calculator
+https://ceph.com/pgcalc/] online. From Ceph Nautilus onward, you can change the
+number of PGs footnoteref:[placement_groups,Placement Groups
+{cephdocs-url}/rados/operations/placement-groups/] after the setup.
+
+In addition to manual adjustment, the PG autoscaler
+footnoteref:[autoscaler,Automated Scaling
+{cephdocs-url}/rados/operations/placement-groups/#automated-scaling] can
+automatically scale the PG count for a pool in the background.
You can create pools through command line or on the GUI on each PVE host under
**Ceph -> Pools**.
mark the checkbox "Add storages" in the GUI or use the command line option
'--add_storages' at pool creation.
+.Base Options
+Name:: The name of the pool. This must be unique and can't be changed afterwards.
+Size:: The number of replicas per object. Ceph always tries to have this many
+copies of an object. Default: `3`.
+PG Autoscale Mode:: The automatic PG scaling mode footnoteref:[autoscaler] of
+the pool. If set to `warn`, it produces a warning message when a pool
+has a non-optimal PG count. Default: `warn`.
+Add as Storage:: Configure a VM or container storage using the new pool.
+Default: `true`.
+
+.Advanced Options
+Min. Size:: The minimum number of replicas per object. Ceph will reject I/O on
+the pool if a PG has less than this many replicas. Default: `2`.
+Crush Rule:: The rule to use for mapping object placement in the cluster. These
+rules define how data is placed within the cluster. See
+xref:pve_ceph_device_classes[Ceph CRUSH & device classes] for information on
+device-based rules.
+# of PGs:: The number of placement groups footnoteref:[placement_groups] that
+the pool should have at the beginning. Default: `128`.
+Traget Size:: The estimated amount of data expected in the pool. The PG
+autoscaler uses this size to estimate the optimal PG count.
+Target Size Ratio:: The ratio of data that is expected in the pool. The PG
+autoscaler uses the ratio relative to other ratio sets. It takes precedence
+over the `target size` if both are set.
+Min. # of PGs:: The minimum number of placement groups. This setting is used to
+fine-tune the lower bound of the PG count for that pool. The PG autoscaler
+will not merge PGs below this threshold.
+
Further information on Ceph pool handling can be found in the Ceph pool
operation footnote:[Ceph pool operation
-https://docs.ceph.com/docs/{ceph_codename}/rados/operations/pools/]
+{cephdocs-url}/rados/operations/pools/]
manual.
OSDs, buckets (device locations) and rulesets (data replication) for pools.
NOTE: Further information can be found in the Ceph documentation, under the
-section CRUSH map footnote:[CRUSH map https://docs.ceph.com/docs/{ceph_codename}/rados/operations/crush-map/].
+section CRUSH map footnote:[CRUSH map {cephdocs-url}/rados/operations/crush-map/].
This map can be altered to reflect different replication hierarchies. The object
replicas can be separated (eg. failure domains), while maintaining the desired
running, but this is normally only useful for a high count on parallel clients,
as else the `MDS` seldom is the bottleneck. If you want to set this up please
refer to the ceph documentation. footnote:[Configuring multiple active MDS
-daemons https://docs.ceph.com/docs/{ceph_codename}/cephfs/multimds/]
+daemons {cephdocs-url}/cephfs/multimds/]
[[pveceph_fs_create]]
Create CephFS
`'cephfs_metadata'' with one quarter of the data pools placement groups (`32`).
Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
Ceph documentation for more information regarding a fitting placement group
-number (`pg_num`) for your setup footnote:[Ceph Placement Groups
-https://docs.ceph.com/docs/{ceph_codename}/rados/operations/placement-groups/].
+number (`pg_num`) for your setup footnoteref:[placement_groups].
Additionally, the `'--add-storage'' parameter will add the CephFS to the {pve}
-storage configuration after it was created successfully.
+storage configuration after it has been created successfully.
Destroy CephFS
~~~~~~~~~~~~~~
cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
the objects and uses checksums to ensure data integrity. If a running scrub
interferes with business (performance) needs, you can adjust the time when
-scrubs footnote:[Ceph scrubbing https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing]
+scrubs footnote:[Ceph scrubbing {cephdocs-url}/rados/configuration/osd-config-ref/#scrubbing]
are executed.
Ceph monitoring and troubleshooting
-----------------------------------
-A good start is to continuosly monitor the ceph health from the start of
+A good start is to continuously monitor the ceph health from the start of
initial deployment. Either through the ceph tools itself, but also by accessing
the status through the {pve} link:api-viewer/index.html[API].
To get a more detailed view, every ceph service has a log file under
`/var/log/ceph/` and if there is not enough detail, the log level can be
-adjusted footnote:[Ceph log and debugging https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/log-and-debug/].
+adjusted footnote:[Ceph log and debugging {cephdocs-url}/rados/troubleshooting/log-and-debug/].
You can find more information about troubleshooting
-footnote:[Ceph troubleshooting https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/]
+footnote:[Ceph troubleshooting {cephdocs-url}/rados/troubleshooting/]
a Ceph cluster on the official website.
projects / pve-docs.git / blobdiff