10 pveceph - Manage Ceph Services on Proxmox VE Nodes
15 include::pveceph.1-synopsis.adoc[]
21 Manage Ceph Services on Proxmox VE Nodes
22 ========================================
26 [thumbnail="gui-ceph-status.png"]
28 {pve} unifies your compute and storage systems, i.e. you can use the
29 same physical nodes within a cluster for both computing (processing
30 VMs and containers) and replicated storage. The traditional silos of
31 compute and storage resources can be wrapped up into a single
32 hyper-converged appliance. Separate storage networks (SANs) and
33 connections via network (NAS) disappear. With the integration of Ceph,
34 an open source software-defined storage platform, {pve} has the
35 ability to run and manage Ceph storage directly on the hypervisor
38 Ceph is a distributed object store and file system designed to provide
39 excellent performance, reliability and scalability. For smaller
40 deployments, it is possible to install a Ceph server for RADOS Block
41 Devices (RBD) directly on your {pve} cluster nodes, see
42 xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
43 hardware has plenty of CPU power and RAM, so running storage services
44 and VMs on the same node is possible.
46 To simplify management, we provide 'pveceph' - a tool to install and
47 manage {ceph} services on {pve} nodes.
53 To build a Proxmox Ceph Cluster there should be at least three (preferably)
54 identical servers for the setup.
56 A 10Gb network, exclusively used for Ceph, is recommmended. A meshed
57 network setup is also an option if there are no 10Gb switches
58 available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] .
60 Check also the recommendations from
61 http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website].
64 Installation of Ceph Packages
65 -----------------------------
67 On each node run the installation script as follows:
74 This sets up an `apt` package repository in
75 `/etc/apt/sources.list.d/ceph.list` and installs the required software.
78 Creating initial Ceph configuration
79 -----------------------------------
81 [thumbnail="gui-ceph-config.png"]
83 After installation of packages, you need to create an initial Ceph
84 configuration on just one node, based on your network (`10.10.10.0/24`
85 in the following example) dedicated for Ceph:
89 pveceph init --network 10.10.10.0/24
92 This creates an initial config at `/etc/pve/ceph.conf`. That file is
93 automatically distributed to all {pve} nodes by using
94 xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link
95 from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run
96 Ceph commands without the need to specify a configuration file.
99 Creating Ceph Monitors
100 ----------------------
102 [thumbnail="gui-ceph-monitor.png"]
104 On each node where a monitor is requested (three monitors are recommended)
105 create it by using the "Ceph" item in the GUI or run.
117 [thumbnail="gui-ceph-osd-status.png"]
119 via GUI or via CLI as follows:
123 pveceph createosd /dev/sd[X]
126 If you want to use a dedicated SSD journal disk:
128 NOTE: In order to use a dedicated journal disk (SSD), the disk needs
129 to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
130 partition table. You can create this with `gdisk /dev/sd(x)`. If there
131 is no GPT, you cannot select the disk as journal. Currently the
132 journal size is fixed to 5 GB.
136 pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X]
139 Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD
144 pveceph createosd /dev/sdf -journal_dev /dev/sdb
147 This partitions the disk (data and journal partition), creates
148 filesystems and starts the OSD, afterwards it is running and fully
149 functional. Please create at least 12 OSDs, distributed among your
150 nodes (4 OSDs on each node).
152 It should be noted that this command refuses to initialize disk when
153 it detects existing data. So if you want to overwrite a disk you
154 should remove existing data first. You can do that using:
158 ceph-disk zap /dev/sd[X]
161 You can create OSDs containing both journal and data partitions or you
162 can place the journal on a dedicated SSD. Using a SSD journal disk is
163 highly recommended if you expect good performance.
169 [thumbnail="gui-ceph-pools.png"]
171 The standard installation creates per default the pool 'rbd',
172 additional pools can be created via GUI.
178 [thumbnail="gui-ceph-log.png"]
180 You can then configure {pve} to use such pools to store VM or
181 Container images. Simply use the GUI too add a new `RBD` storage (see
182 section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
184 You also need to copy the keyring to a predefined location.
186 NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
187 the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
188 `my-ceph-storage` in the following example:
192 mkdir /etc/pve/priv/ceph
193 cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
198 include::pve-copyright.adoc[]