pveceph.adoc

   1 [[chapter_pveceph]]
   2 ifdef::manvolnum[]
   3 pveceph(1)
   4 ==========
   5 :pve-toplevel:
   6
   7 NAME
   8 ----
   9
  10 pveceph - Manage Ceph Services on Proxmox VE Nodes
  11
  12 SYNOPSIS
  13 --------
  14
  15 include::pveceph.1-synopsis.adoc[]
  16
  17 DESCRIPTION
  18 -----------
  19 endif::manvolnum[]
  20 ifndef::manvolnum[]
  21 Manage Ceph Services on Proxmox VE Nodes
  22 ========================================
  23 :pve-toplevel:
  24 endif::manvolnum[]
  25
  26 {pve} unifies your compute and storage systems, i.e. you can use the
  27 same physical nodes within a cluster for both computing (processing
  28 VMs and containers) and replicated storage. The traditional silos of
  29 compute and storage resources can be wrapped up into a single
  30 hyper-converged appliance. Separate storage networks (SANs) and
  31 connections via network (NAS) disappear. With the integration of Ceph,
  32 an open source software-defined storage platform, {pve} has the
  33 ability to run and manage Ceph storage directly on the hypervisor
  34 nodes.
  35
  36 Ceph is a distributed object store and file system designed to provide
  37 excellent performance, reliability and scalability. For smaller
  38 deployments, it is possible to install a Ceph server for RADOS Block
  39 Devices (RBD) directly on your {pve} cluster nodes, see
  40 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
  41 hardware has plenty of CPU power and RAM, so running storage services
  42 and VMs on the same node is possible.
  43
  44 To simplify management, we provide 'pveceph' - a tool to install and
  45 manage {ceph} services on {pve} nodes.
  46
  47
  48 Precondition
  49 ------------
  50
  51 To build a Proxmox Ceph Cluster there should be at least three (preferably)
  52 identical servers for the setup.
  53
  54 A 10Gb network, exclusively used for Ceph, is recommmended. A meshed
  55 network setup is also an option if there are no 10Gb switches
  56 available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] .
  57
  58 Check also the recommendations from
  59 http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website].
  60
  61
  62 Installation of Ceph Packages
  63 -----------------------------
  64
  65 On each node run the installation script as follows:
  66
  67 [source,bash]
  68 ----
  69 pveceph install
  70 ----
  71
  72 This sets up an `apt` package repository in
  73 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
  74
  75
  76 Creating initial Ceph configuration
  77 -----------------------------------
  78
  79 After installation of packages, you need to create an initial Ceph
  80 configuration on just one node, based on your network (`10.10.10.0/24`
  81 in the following example) dedicated for Ceph:
  82
  83 [source,bash]
  84 ----
  85 pveceph init --network 10.10.10.0/24
  86 ----
  87
  88 This creates an initial config at `/etc/pve/ceph.conf`. That file is
  89 automatically distributed to all {pve} nodes by using
  90 xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link
  91 from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run
  92 Ceph commands without the need to specify a configuration file.
  93
  94
  95 Creating Ceph Monitors
  96 ----------------------
  97
  98 On each node where a monitor is requested (three monitors are recommended)
  99 create it by using the "Ceph" item in the GUI or run.
 100
 101
 102 [source,bash]
 103 ----
 104 pveceph createmon
 105 ----
 106
 107
 108 Creating Ceph OSDs
 109 ------------------
 110
 111 via GUI or via CLI as follows:
 112
 113 [source,bash]
 114 ----
 115 pveceph createosd /dev/sd[X]
 116 ----
 117
 118 If you want to use a dedicated SSD journal disk:
 119
 120 NOTE: In order to use a dedicated journal disk (SSD), the disk needs
 121 to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
 122 partition table. You can create this with `gdisk /dev/sd(x)`. If there
 123 is no GPT, you cannot select the disk as journal. Currently the
 124 journal size is fixed to 5 GB.
 125
 126 [source,bash]
 127 ----
 128 pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X]
 129 ----
 130
 131 Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD
 132 journal disk.
 133
 134 [source,bash]
 135 ----
 136 pveceph createosd /dev/sdf -journal_dev /dev/sdb
 137 ----
 138
 139 This partitions the disk (data and journal partition), creates
 140 filesystems and starts the OSD, afterwards it is running and fully
 141 functional. Please create at least 12 OSDs, distributed among your
 142 nodes (4 OSDs on each node).
 143
 144 It should be noted that this command refuses to initialize disk when
 145 it detects existing data. So if you want to overwrite a disk you
 146 should remove existing data first. You can do that using:
 147
 148 [source,bash]
 149 ----
 150 ceph-disk zap /dev/sd[X]
 151 ----
 152
 153 You can create OSDs containing both journal and data partitions or you
 154 can place the journal on a dedicated SSD. Using a SSD journal disk is
 155 highly recommended if you expect good performance.
 156
 157
 158 Ceph Pools
 159 ----------
 160
 161 The standard installation creates per default the pool 'rbd',
 162 additional pools can be created via GUI.
 163
 164
 165 Ceph Client
 166 -----------
 167
 168 You can then configure {pve} to use such pools to store VM or
 169 Container images. Simply use the GUI too add a new `RBD` storage (see
 170 section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
 171
 172 You also need to copy the keyring to a predefined location.
 173
 174 NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
 175 the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
 176 `my-ceph-storage` in the following example:
 177
 178 [source,bash]
 179 ----
 180 mkdir /etc/pve/priv/ceph
 181 cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
 182 ----
 183
 184
 185 ifdef::manvolnum[]
 186 include::pve-copyright.adoc[]
 187 endif::manvolnum[]