X-Git-Url: https://git.proxmox.com/?p=pve-docs.git;a=blobdiff_plain;f=pveceph.adoc;h=8dc85680ded076094afd9d7afb724f5ccd1761e7;hp=d01ae462d0c49a4b2153de6060fe6259d2efee82;hb=31df4fee0443eb4e00fdf2a6cbfc88c2789f5882;hpb=af6f59f49f55a42b15cc52c49497baf80aed5c0b diff --git a/pveceph.adoc b/pveceph.adoc index d01ae46..8dc8568 100644 --- a/pveceph.adoc +++ b/pveceph.adoc @@ -18,8 +18,8 @@ DESCRIPTION ----------- endif::manvolnum[] ifndef::manvolnum[] -Manage Ceph Services on Proxmox VE Nodes -======================================== +Deploy Hyper-Converged Ceph Cluster +=================================== :pve-toplevel: endif::manvolnum[] @@ -86,9 +86,16 @@ provide enough resources for stable and durable Ceph performance. .Memory Especially in a hyper-converged setup, the memory consumption needs to be carefully monitored. In addition to the intended workload from virtual machines -and container, Ceph needs enough memory available to provide good and stable -performance. As a rule of thumb, for roughly 1 TiB of data, 1 GiB of memory -will be used by an OSD. OSD caching will use additional memory. +and containers, Ceph needs enough memory available to provide excellent and +stable performance. + +As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used +by an OSD. Especially during recovery, rebalancing or backfilling. + +The daemon itself will use additional memory. The Bluestore backend of the +daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the +legacy Filestore backend uses the OS page cache and the memory consumption is +generally related to PGs of an OSD daemon. .Network We recommend a network bandwidth of at least 10 GbE or more, which is used @@ -101,7 +108,7 @@ services on the same network and may even break the {pve} cluster stack. Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate -10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwith +10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwidth will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or even 100 GBps are possible. @@ -286,7 +293,7 @@ monitor the cluster. Since the Ceph luminous release at least one ceph-mgr footnote:[Ceph Manager https://docs.ceph.com/docs/{ceph_codename}/mgr/] daemon is required. -[i[pveceph_create_mgr]] +[[pveceph_create_mgr]] Create Manager ~~~~~~~~~~~~~~ @@ -378,7 +385,7 @@ pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z] ---- You can directly choose the size for those with the '-db_size' and '-wal_size' -paremeters respectively. If they are not given the following values (in order) +parameters respectively. If they are not given the following values (in order) will be used: * bluestore_block_{db,wal}_size from ceph configuration... @@ -758,21 +765,23 @@ pveceph osd destroy Replace the old disk with the new one and use the same procedure as described in xref:pve_ceph_osd_create[Create OSDs]. -Run fstrim (discard) -~~~~~~~~~~~~~~~~~~~~ +Trim/Discard +~~~~~~~~~~~~ It is a good measure to run 'fstrim' (discard) regularly on VMs or containers. This releases data blocks that the filesystem isn’t using anymore. It reduces -data usage and the resource load. +data usage and resource load. Most modern operating systems issue such discard +commands to their disks regularly. You only need to ensure that the Virtual +Machines enable the xref:qm_hard_disk_discard[disk discard option]. [[pveceph_scrub]] Scrub & Deep Scrub ~~~~~~~~~~~~~~~~~~ Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every object in a PG for its health. There are two forms of Scrubbing, daily -(metadata compare) and weekly. The weekly reads the objects and uses checksums -to ensure data integrity. If a running scrub interferes with business needs, -you can adjust the time when scrubs footnote:[Ceph scrubbing -https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing] +cheap metadata checks and weekly deep data checks. The weekly deep scrub reads +the objects and uses checksums to ensure data integrity. If a running scrub +interferes with business (performance) needs, you can adjust the time when +scrubs footnote:[Ceph scrubbing https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing] are executed.