pveceph.adoc

   1 [[chapter_pveceph]]
   2 ifdef::manvolnum[]
   3 pveceph(1)
   4 ==========
   5 :pve-toplevel:
   6
   7 NAME
   8 ----
   9
  10 pveceph - Manage Ceph Services on Proxmox VE Nodes
  11
  12 SYNOPSIS
  13 --------
  14
  15 include::pveceph.1-synopsis.adoc[]
  16
  17 DESCRIPTION
  18 -----------
  19 endif::manvolnum[]
  20 ifndef::manvolnum[]
  21 pveceph - Manage Ceph Services on Proxmox VE Nodes
  22 ==================================================
  23 endif::manvolnum[]
  24
  25 {pve} unifies your compute and storage systems, i.e. you can use the
  26 same physical nodes within a cluster for both computing (processing
  27 VMs and containers) and replicated storage. The traditional silos of
  28 compute and storage resources can be wrapped up into a single
  29 hyper-converged appliance. Separate storage networks (SANs) and
  30 connections via network (NAS) disappear. With the integration of Ceph,
  31 an open source software-defined storage platform, {pve} has the
  32 ability to run and manage Ceph storage directly on the hypervisor
  33 nodes.
  34
  35 Ceph is a distributed object store and file system designed to provide
  36 excellent performance, reliability and scalability. For smaller
  37 deployments, it is possible to install a Ceph server for RADOS Block
  38 Devices (RBD) directly on your {pve} cluster nodes, see
  39 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
  40 hardware has plenty of CPU power and RAM, so running storage services
  41 and VMs on the same node is possible.
  42
  43 To simplify management, we provide 'pveceph' - a tool to install and
  44 manage {ceph} services on {pve} nodes.
  45
  46
  47 Precondition
  48 ------------
  49
  50 To build a Proxmox Ceph Cluster there should be at least three (preferably)
  51 identical servers for the setup.
  52
  53 A 10Gb network, exclusively used for Ceph, is recommmended. A meshed
  54 network setup is also an option if there are no 10Gb switches
  55 available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] .
  56
  57 Check also the recommendations from
  58 http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website].
  59
  60
  61 Installation of Ceph Packages
  62 -----------------------------
  63
  64 On each node run the installation script as follows:
  65
  66 [source,bash]
  67 ----
  68 pveceph install
  69 ----
  70
  71 This sets up an `apt` package repository in
  72 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
  73
  74
  75 Creating initial Ceph configuration
  76 -----------------------------------
  77
  78 After installation of packages, you need to create an initial Ceph
  79 configuration on just one node, based on your network (`10.10.10.0/24`
  80 in the following example) dedicated for Ceph:
  81
  82 [source,bash]
  83 ----
  84 pveceph init --network 10.10.10.0/24
  85 ----
  86
  87 This creates an initial config at `/etc/pve/ceph.conf`. That file is
  88 automatically distributed to all {pve} nodes by using
  89 xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link
  90 from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run
  91 Ceph commands without the need to specify a configuration file.
  92
  93
  94 Creating Ceph Monitors
  95 ----------------------
  96
  97 On each node where a monitor is requested (three monitors are recommended)
  98 create it by using the "Ceph" item in the GUI or run.
  99
 100
 101 [source,bash]
 102 ----
 103 pveceph createmon
 104 ----
 105
 106
 107 Creating Ceph OSDs
 108 ------------------
 109
 110 via GUI or via CLI as follows:
 111
 112 [source,bash]
 113 ----
 114 pveceph createosd /dev/sd[X]
 115 ----
 116
 117 If you want to use a dedicated SSD journal disk:
 118
 119 NOTE: In order to use a dedicated journal disk (SSD), the disk needs
 120 to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
 121 partition table. You can create this with `gdisk /dev/sd(x)`. If there
 122 is no GPT, you cannot select the disk as journal. Currently the
 123 journal size is fixed to 5 GB.
 124
 125 [source,bash]
 126 ----
 127 pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X]
 128 ----
 129
 130 Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD
 131 journal disk.
 132
 133 [source,bash]
 134 ----
 135 pveceph createosd /dev/sdf -journal_dev /dev/sdb
 136 ----
 137
 138 This partitions the disk (data and journal partition), creates
 139 filesystems and starts the OSD, afterwards it is running and fully
 140 functional. Please create at least 12 OSDs, distributed among your
 141 nodes (4 OSDs on each node).
 142
 143 It should be noted that this command refuses to initialize disk when
 144 it detects existing data. So if you want to overwrite a disk you
 145 should remove existing data first. You can do that using:
 146
 147 [source,bash]
 148 ----
 149 ceph-disk zap /dev/sd[X]
 150 ----
 151
 152 You can create OSDs containing both journal and data partitions or you
 153 can place the journal on a dedicated SSD. Using a SSD journal disk is
 154 highly recommended if you expect good performance.
 155
 156
 157 Ceph Pools
 158 ----------
 159
 160 The standard installation creates per default the pool 'rbd',
 161 additional pools can be created via GUI.
 162
 163
 164 Ceph Client
 165 -----------
 166
 167 You can then configure {pve} to use such pools to store VM or
 168 Container images. Simply use the GUI too add a new `RBD` storage (see
 169 section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
 170
 171 You also need to copy the keyring to a predefined location.
 172
 173 NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
 174 the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
 175 `my-ceph-storage` in the following example:
 176
 177 [source,bash]
 178 ----
 179 mkdir /etc/pve/priv/ceph
 180 cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
 181 ----
 182
 183
 184 ifdef::manvolnum[]
 185 include::pve-copyright.adoc[]
 186 endif::manvolnum[]