fix wiki-special link to "USB installation"

[pve-docs.git] / pveceph.adoc

diff --git a/pveceph.adoc b/pveceph.adoc
index e6643952dfdc25e88cca1737615a71185de9fb78..8dc85680ded076094afd9d7afb724f5ccd1761e7 100644 (file)
--- a/pveceph.adoc
+++ b/pveceph.adoc
@@ -18,8 +18,8 @@ DESCRIPTION
 -----------
 endif::manvolnum[]
 ifndef::manvolnum[]
 -----------
 endif::manvolnum[]
 ifndef::manvolnum[]

-Manage Ceph Services on Proxmox VE Nodes
-========================================
+Deploy Hyper-Converged Ceph Cluster
+===================================

 :pve-toplevel:
 endif::manvolnum[]
 
 :pve-toplevel:
 endif::manvolnum[]
 


@@ -58,15 +58,15 @@ and VMs on the same node is possible.
 To simplify management, we provide 'pveceph' - a tool to install and
 manage {ceph} services on {pve} nodes.
 
 To simplify management, we provide 'pveceph' - a tool to install and
 manage {ceph} services on {pve} nodes.
 

-.Ceph consists of a couple of Daemons footnote:[Ceph intro http://docs.ceph.com/docs/luminous/start/intro/], for use as a RBD storage:
+.Ceph consists of a couple of Daemons footnote:[Ceph intro https://docs.ceph.com/docs/{ceph_codename}/start/intro/], for use as a RBD storage:

 - Ceph Monitor (ceph-mon)
 - Ceph Manager (ceph-mgr)
 - Ceph OSD (ceph-osd; Object Storage Daemon)
 
 TIP: We highly recommend to get familiar with Ceph's architecture
 - Ceph Monitor (ceph-mon)
 - Ceph Manager (ceph-mgr)
 - Ceph OSD (ceph-osd; Object Storage Daemon)
 
 TIP: We highly recommend to get familiar with Ceph's architecture

-footnote:[Ceph architecture http://docs.ceph.com/docs/luminous/architecture/]
+footnote:[Ceph architecture https://docs.ceph.com/docs/{ceph_codename}/architecture/]

 and vocabulary
 and vocabulary

-footnote:[Ceph glossary http://docs.ceph.com/docs/luminous/glossary].
+footnote:[Ceph glossary https://docs.ceph.com/docs/{ceph_codename}/glossary].

 
 
 Precondition
 
 
 Precondition


@@ -76,7 +76,7 @@ To build a hyper-converged Proxmox + Ceph Cluster there should be at least
 three (preferably) identical servers for the setup.
 
 Check also the recommendations from
 three (preferably) identical servers for the setup.
 
 Check also the recommendations from

-http://docs.ceph.com/docs/luminous/start/hardware-recommendations/[Ceph's website].
+https://docs.ceph.com/docs/{ceph_codename}/start/hardware-recommendations/[Ceph's website].

 
 .CPU
 Higher CPU core frequency reduce latency and should be preferred. As a simple
 
 .CPU
 Higher CPU core frequency reduce latency and should be preferred. As a simple


@@ -86,9 +86,16 @@ provide enough resources for stable and durable Ceph performance.
 .Memory
 Especially in a hyper-converged setup, the memory consumption needs to be
 carefully monitored. In addition to the intended workload from virtual machines
 .Memory
 Especially in a hyper-converged setup, the memory consumption needs to be
 carefully monitored. In addition to the intended workload from virtual machines

-and container, Ceph needs enough memory available to provide good and stable
-performance. As a rule of thumb, for roughly 1 TiB of data, 1 GiB of memory
-will be used by an OSD. OSD caching will use additional memory.
+and containers, Ceph needs enough memory available to provide excellent and
+stable performance.
+
+As a rule of thumb, for roughly **1 TiB of data, 1 GiB of memory** will be used
+by an OSD. Especially during recovery, rebalancing or backfilling.
+
+The daemon itself will use additional memory. The Bluestore backend of the
+daemon requires by default **3-5 GiB of memory** (adjustable). In contrast, the
+legacy Filestore backend uses the OS page cache and the memory consumption is
+generally related to PGs of an OSD daemon.

 
 .Network
 We recommend a network bandwidth of at least 10 GbE or more, which is used
 
 .Network
 We recommend a network bandwidth of at least 10 GbE or more, which is used


@@ -101,7 +108,7 @@ services on the same network and may even break the {pve} cluster stack.
 
 Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
 link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate
 
 Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
 link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate

-10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwith
+10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwidth

 will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or
 even 100 GBps are possible.
 
 will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or
 even 100 GBps are possible.
 


@@ -212,8 +219,8 @@ This sets up an `apt` package repository in
 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
 
 
 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
 
 

-Creating initial Ceph configuration
------------------------------------
+Create initial Ceph configuration
+---------------------------------

 
 [thumbnail="screenshot/gui-ceph-config.png"]
 
 
 [thumbnail="screenshot/gui-ceph-config.png"]
 


@@ -234,19 +241,23 @@ configuration file.
 
 
 [[pve_ceph_monitors]]
 
 
 [[pve_ceph_monitors]]

-Creating Ceph Monitors
-----------------------
-
-[thumbnail="screenshot/gui-ceph-monitor.png"]
-
+Ceph Monitor
+-----------

 The Ceph Monitor (MON)
 The Ceph Monitor (MON)

-footnote:[Ceph Monitor http://docs.ceph.com/docs/luminous/start/intro/]
+footnote:[Ceph Monitor https://docs.ceph.com/docs/{ceph_codename}/start/intro/]

 maintains a master copy of the cluster map. For high availability you need to
 have at least 3 monitors. One monitor will already be installed if you
 used the installation wizard. You won't need more than 3 monitors as long
 as your cluster is small to midsize, only really large clusters will
 need more than that.
 
 maintains a master copy of the cluster map. For high availability you need to
 have at least 3 monitors. One monitor will already be installed if you
 used the installation wizard. You won't need more than 3 monitors as long
 as your cluster is small to midsize, only really large clusters will
 need more than that.
 

+
+[[pveceph_create_mon]]
+Create Monitors
+~~~~~~~~~~~~~~~
+
+[thumbnail="screenshot/gui-ceph-monitor.png"]
+

 On each node where you want to place a monitor (three monitors are recommended),
 create it by using the 'Ceph -> Monitor' tab in the GUI or run.
 
 On each node where you want to place a monitor (three monitors are recommended),
 create it by using the 'Ceph -> Monitor' tab in the GUI or run.
 


@@ -256,12 +267,9 @@ create it by using the 'Ceph -> Monitor' tab in the GUI or run.
 pveceph mon create
 ----
 
 pveceph mon create
 ----
 

-This will also install the needed Ceph Manager ('ceph-mgr') by default. If you
-do not want to install a manager, specify the '-exclude-manager' option.
-
-
-Destroying Ceph Monitor
-----------------------
+[[pveceph_destroy_mon]]
+Destroy Monitors
+~~~~~~~~~~~~~~~~

 
 To remove a Ceph Monitor via the GUI first select a node in the tree view and
 go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**
 
 To remove a Ceph Monitor via the GUI first select a node in the tree view and
 go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**


@@ -278,26 +286,31 @@ NOTE: At least three Monitors are needed for quorum.
 
 
 [[pve_ceph_manager]]
 
 
 [[pve_ceph_manager]]

-Creating Ceph Manager
-----------------------
+Ceph Manager
+------------
+The Manager daemon runs alongside the monitors. It provides an interface to
+monitor the cluster. Since the Ceph luminous release at least one ceph-mgr
+footnote:[Ceph Manager https://docs.ceph.com/docs/{ceph_codename}/mgr/] daemon is
+required.

 
 

-The Manager daemon runs alongside the monitors, providing an interface for
-monitoring the cluster. Since the Ceph luminous release the
-ceph-mgr footnote:[Ceph Manager http://docs.ceph.com/docs/luminous/mgr/] daemon
-is required. During monitor installation the ceph manager will be installed as
-well.
+[[pveceph_create_mgr]]
+Create Manager
+~~~~~~~~~~~~~~

 
 

-NOTE: It is recommended to install the Ceph Manager on the monitor nodes. For
-high availability install more then one manager.
+Multiple Managers can be installed, but at any time only one Manager is active.

 
 [source,bash]
 ----
 pveceph mgr create
 ----
 
 
 [source,bash]
 ----
 pveceph mgr create
 ----
 

+NOTE: It is recommended to install the Ceph Manager on the monitor nodes. For
+high availability install more then one manager.

 
 

-Destroying Ceph Manager
-----------------------
+
+[[pveceph_destroy_mgr]]
+Destroy Manager
+~~~~~~~~~~~~~~~

 
 To remove a Ceph Manager via the GUI first select a node in the tree view and
 go to the **Ceph -> Monitor** panel. Select the Manager and click the
 
 To remove a Ceph Manager via the GUI first select a node in the tree view and
 go to the **Ceph -> Monitor** panel. Select the Manager and click the


@@ -315,8 +328,16 @@ the cluster status or usage require a running Manager.
 
 
 [[pve_ceph_osds]]
 
 
 [[pve_ceph_osds]]

-Creating Ceph OSDs
-------------------
+Ceph OSDs
+---------
+Ceph **O**bject **S**torage **D**aemons are storing objects for Ceph over the
+network. It is recommended to use one OSD per physical disk.
+
+NOTE: By default an object is 4 MiB in size.
+
+[[pve_ceph_osd_create]]
+Create OSDs
+~~~~~~~~~~~

 
 [thumbnail="screenshot/gui-ceph-osd-status.png"]
 
 
 [thumbnail="screenshot/gui-ceph-osd-status.png"]
 


@@ -327,8 +348,8 @@ via GUI or via CLI as follows:
 pveceph osd create /dev/sd[X]
 ----
 
 pveceph osd create /dev/sd[X]
 ----
 

-TIP: We recommend a Ceph cluster size, starting with 12 OSDs, distributed evenly
-among your, at least three nodes (4 OSDs on each node).
+TIP: We recommend a Ceph cluster size, starting with 12 OSDs, distributed
+evenly among your, at least three nodes (4 OSDs on each node).

 
 If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot
 sector and any OSD leftover the following command should be sufficient.
 
 If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot
 sector and any OSD leftover the following command should be sufficient.


@@ -340,12 +361,11 @@ ceph-volume lvm zap /dev/sd[X] --destroy
 
 WARNING: The above command will destroy data on the disk!
 
 
 WARNING: The above command will destroy data on the disk!
 

-Ceph Bluestore
-~~~~~~~~~~~~~~
+.Ceph Bluestore

 
 Starting with the Ceph Kraken release, a new Ceph OSD storage type was
 introduced, the so called Bluestore
 
 Starting with the Ceph Kraken release, a new Ceph OSD storage type was
 introduced, the so called Bluestore

-footnote:[Ceph Bluestore http://ceph.com/community/new-luminous-bluestore/].
+footnote:[Ceph Bluestore https://ceph.com/community/new-luminous-bluestore/].

 This is the default when creating OSDs since Ceph Luminous.
 
 [source,bash]
 This is the default when creating OSDs since Ceph Luminous.
 
 [source,bash]


@@ -356,8 +376,8 @@ pveceph osd create /dev/sd[X]
 .Block.db and block.wal
 
 If you want to use a separate DB/WAL device for your OSDs, you can specify it
 .Block.db and block.wal
 
 If you want to use a separate DB/WAL device for your OSDs, you can specify it

-through the '-db_dev' and '-wal_dev' options. The WAL is placed with the DB, if not
-specified separately.
+through the '-db_dev' and '-wal_dev' options. The WAL is placed with the DB, if
+not specified separately.

 
 [source,bash]
 ----
 
 [source,bash]
 ----


@@ -365,7 +385,7 @@ pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]
 ----
 
 You can directly choose the size for those with the '-db_size' and '-wal_size'
 ----
 
 You can directly choose the size for those with the '-db_size' and '-wal_size'

-paremeters respectively. If they are not given the following values (in order)
+parameters respectively. If they are not given the following values (in order)

 will be used:
 
 * bluestore_block_{db,wal}_size from ceph configuration...
 will be used:
 
 * bluestore_block_{db,wal}_size from ceph configuration...


@@ -380,8 +400,7 @@ internal journal or write-ahead log. It is recommended to use a fast SSD or
 NVRAM for better performance.
 
 
 NVRAM for better performance.
 
 

-Ceph Filestore
-~~~~~~~~~~~~~~
+.Ceph Filestore

 
 Before Ceph Luminous, Filestore was used as default storage type for Ceph OSDs.
 Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
 
 Before Ceph Luminous, Filestore was used as default storage type for Ceph OSDs.
 Starting with Ceph Nautilus, {pve} does not support creating such OSDs with


@@ -393,8 +412,9 @@ Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
 ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
 ----
 
 ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
 ----
 

-Destroying Ceph OSDs
---------------------
+[[pve_ceph_osd_destroy]]
+Destroy OSDs
+~~~~~~~~~~~~

 
 To remove an OSD via the GUI first select a {PVE} node in the tree view and go
 to the **Ceph -> OSD** panel. Select the OSD to destroy. Next click the **OUT**
 
 To remove an OSD via the GUI first select a {PVE} node in the tree view and go
 to the **Ceph -> OSD** panel. Select the OSD to destroy. Next click the **OUT**


@@ -422,14 +442,17 @@ WARNING: The above command will destroy data on the disk!
 
 
 [[pve_ceph_pools]]
 
 
 [[pve_ceph_pools]]

-Creating Ceph Pools
--------------------
-
-[thumbnail="screenshot/gui-ceph-pools.png"]
-
+Ceph Pools
+----------

 A pool is a logical group for storing objects. It holds **P**lacement
 **G**roups (`PG`, `pg_num`), a collection of objects.
 
 A pool is a logical group for storing objects. It holds **P**lacement
 **G**roups (`PG`, `pg_num`), a collection of objects.
 

+
+Create Pools
+~~~~~~~~~~~~
+
+[thumbnail="screenshot/gui-ceph-pools.png"]
+

 When no options are given, we set a default of **128 PGs**, a **size of 3
 replicas** and a **min_size of 2 replicas** for serving objects in a degraded
 state.
 When no options are given, we set a default of **128 PGs**, a **size of 3
 replicas** and a **min_size of 2 replicas** for serving objects in a degraded
 state.


@@ -439,7 +462,7 @@ NOTE: The default number of PGs works for 2-5 disks. Ceph throws a
 
 It is advised to calculate the PG number depending on your setup, you can find
 the formula and the PG calculator footnote:[PG calculator
 
 It is advised to calculate the PG number depending on your setup, you can find
 the formula and the PG calculator footnote:[PG calculator

-http://ceph.com/pgcalc/] online. While PGs can be increased later on, they can
+https://ceph.com/pgcalc/] online. While PGs can be increased later on, they can

 never be decreased.
 
 
 never be decreased.
 
 


@@ -457,12 +480,12 @@ mark the checkbox "Add storages" in the GUI or use the command line option
 
 Further information on Ceph pool handling can be found in the Ceph pool
 operation footnote:[Ceph pool operation
 
 Further information on Ceph pool handling can be found in the Ceph pool
 operation footnote:[Ceph pool operation

-http://docs.ceph.com/docs/luminous/rados/operations/pools/]
+https://docs.ceph.com/docs/{ceph_codename}/rados/operations/pools/]

 manual.
 
 
 manual.
 
 

-Destroying Ceph Pools
----------------------
+Destroy Pools
+~~~~~~~~~~~~~

 
 To destroy a pool via the GUI select a node in the tree view and go to the
 **Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**
 
 To destroy a pool via the GUI select a node in the tree view and go to the
 **Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**


@@ -490,7 +513,7 @@ advantage that no central index service is needed. CRUSH works with a map of
 OSDs, buckets (device locations) and rulesets (data replication) for pools.
 
 NOTE: Further information can be found in the Ceph documentation, under the
 OSDs, buckets (device locations) and rulesets (data replication) for pools.
 
 NOTE: Further information can be found in the Ceph documentation, under the

-section CRUSH map footnote:[CRUSH map http://docs.ceph.com/docs/luminous/rados/operations/crush-map/].
+section CRUSH map footnote:[CRUSH map https://docs.ceph.com/docs/{ceph_codename}/rados/operations/crush-map/].

 
 This map can be altered to reflect different replication hierarchies. The object
 replicas can be separated (eg. failure domains), while maintaining the desired
 
 This map can be altered to reflect different replication hierarchies. The object
 replicas can be separated (eg. failure domains), while maintaining the desired


@@ -553,8 +576,8 @@ ceph osd pool set <pool-name> crush_rule <rule-name>
 ----
 
 TIP: If the pool already contains objects, all of these have to be moved
 ----
 
 TIP: If the pool already contains objects, all of these have to be moved

-accordingly. Depending on your setup this may introduce a big performance hit on
-your cluster. As an alternative, you can create a new pool and move disks
+accordingly. Depending on your setup this may introduce a big performance hit
+on your cluster. As an alternative, you can create a new pool and move disks

 separately.
 
 
 separately.
 
 


@@ -636,11 +659,11 @@ Since Luminous (12.2.x) you can also have multiple active metadata servers
 running, but this is normally only useful for a high count on parallel clients,
 as else the `MDS` seldom is the bottleneck. If you want to set this up please
 refer to the ceph documentation. footnote:[Configuring multiple active MDS
 running, but this is normally only useful for a high count on parallel clients,
 as else the `MDS` seldom is the bottleneck. If you want to set this up please
 refer to the ceph documentation. footnote:[Configuring multiple active MDS

-daemons http://docs.ceph.com/docs/luminous/cephfs/multimds/]
+daemons https://docs.ceph.com/docs/{ceph_codename}/cephfs/multimds/]

 
 [[pveceph_fs_create]]
 
 [[pveceph_fs_create]]

-Create a CephFS
-~~~~~~~~~~~~~~~
+Create CephFS
+~~~~~~~~~~~~~

 
 With {pve}'s CephFS integration into you can create a CephFS easily over the
 Web GUI, the CLI or an external API interface. Some prerequisites are required
 
 With {pve}'s CephFS integration into you can create a CephFS easily over the
 Web GUI, the CLI or an external API interface. Some prerequisites are required


@@ -668,7 +691,7 @@ This creates a CephFS named `'cephfs'' using a pool for its data named
 Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
 Ceph documentation for more information regarding a fitting placement group
 number (`pg_num`) for your setup footnote:[Ceph Placement Groups
 Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
 Ceph documentation for more information regarding a fitting placement group
 number (`pg_num`) for your setup footnote:[Ceph Placement Groups

-http://docs.ceph.com/docs/luminous/rados/operations/placement-groups/].
+https://docs.ceph.com/docs/{ceph_codename}/rados/operations/placement-groups/].

 Additionally, the `'--add-storage'' parameter will add the CephFS to the {pve}
 storage configuration after it was created successfully.
 
 Additionally, the `'--add-storage'' parameter will add the CephFS to the {pve}
 storage configuration after it was created successfully.
 


@@ -701,6 +724,67 @@ pveceph pool destroy NAME
 ----
 
 
 ----
 
 

+Ceph maintenance
+----------------
+
+Replace OSDs
+~~~~~~~~~~~~
+
+One of the common maintenance tasks in Ceph is to replace a disk of an OSD. If
+a disk is already in a failed state, then you can go ahead and run through the
+steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate those
+copies on the remaining OSDs if possible. This rebalancing will start as soon
+as an OSD failure is detected or an OSD was actively stopped.
+
+NOTE: With the default size/min_size (3/2) of a pool, recovery only starts when
+`size + 1` nodes are available. The reason for this is that the Ceph object
+balancer xref:pve_ceph_device_classes[CRUSH] defaults to a full node as
+`failure domain'.
+
+To replace a still functioning disk, on the GUI go through the steps in
+xref:pve_ceph_osd_destroy[Destroy OSDs]. The only addition is to wait until
+the cluster shows 'HEALTH_OK' before stopping the OSD to destroy it.
+
+On the command line use the following commands.
+----
+ceph osd out osd.<id>
+----
+
+You can check with the command below if the OSD can be safely removed.
+----
+ceph osd safe-to-destroy osd.<id>
+----
+
+Once the above check tells you that it is save to remove the OSD, you can
+continue with following commands.
+----
+systemctl stop ceph-osd@<id>.service
+pveceph osd destroy <id>
+----
+
+Replace the old disk with the new one and use the same procedure as described
+in xref:pve_ceph_osd_create[Create OSDs].
+
+Trim/Discard
+~~~~~~~~~~~~
+It is a good measure to run 'fstrim' (discard) regularly on VMs or containers.
+This releases data blocks that the filesystem isn’t using anymore. It reduces
+data usage and resource load. Most modern operating systems issue such discard
+commands to their disks regularly. You only need to ensure that the Virtual
+Machines enable the xref:qm_hard_disk_discard[disk discard option].
+
+[[pveceph_scrub]]
+Scrub & Deep Scrub
+~~~~~~~~~~~~~~~~~~
+Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every
+object in a PG for its health. There are two forms of Scrubbing, daily
+cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
+the objects and uses checksums to ensure data integrity. If a running scrub
+interferes with business (performance) needs, you can adjust the time when
+scrubs footnote:[Ceph scrubbing https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing]
+are executed.
+
+

 Ceph monitoring and troubleshooting
 -----------------------------------
 A good start is to continuosly monitor the ceph health from the start of
 Ceph monitoring and troubleshooting
 -----------------------------------
 A good start is to continuosly monitor the ceph health from the start of


@@ -721,10 +805,10 @@ pve# ceph -w
 
 To get a more detailed view, every ceph service has a log file under
 `/var/log/ceph/` and if there is not enough detail, the log level can be
 
 To get a more detailed view, every ceph service has a log file under
 `/var/log/ceph/` and if there is not enough detail, the log level can be

-adjusted footnote:[Ceph log and debugging http://docs.ceph.com/docs/luminous/rados/troubleshooting/log-and-debug/].
+adjusted footnote:[Ceph log and debugging https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/log-and-debug/].

 
 You can find more information about troubleshooting
 
 You can find more information about troubleshooting

-footnote:[Ceph troubleshooting http://docs.ceph.com/docs/luminous/rados/troubleshooting/]
+footnote:[Ceph troubleshooting https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/]

 a Ceph cluster on the official website.
 
 
 a Ceph cluster on the official website.