ha: document ignored state

[pve-docs.git] / pveceph.adoc

[[chapter_pveceph]]
ifdef::manvolnum[]
pveceph(1)
==========
:pve-toplevel:

NAME
----

pveceph - Manage Ceph Services on Proxmox VE Nodes

SYNOPSIS
--------

include::pveceph.1-synopsis.adoc[]

DESCRIPTION
-----------
endif::manvolnum[]
ifndef::manvolnum[]
Manage Ceph Services on Proxmox VE Nodes
========================================
:pve-toplevel:
endif::manvolnum[]

[thumbnail="gui-ceph-status.png"]

{pve} unifies your compute and storage systems, i.e. you can use the
same physical nodes within a cluster for both computing (processing
VMs and containers) and replicated storage. The traditional silos of
compute and storage resources can be wrapped up into a single
hyper-converged appliance. Separate storage networks (SANs) and
connections via network (NAS) disappear. With the integration of Ceph,
an open source software-defined storage platform, {pve} has the
ability to run and manage Ceph storage directly on the hypervisor
nodes.

Ceph is a distributed object store and file system designed to provide
excellent performance, reliability and scalability. For smaller
deployments, it is possible to install a Ceph server for RADOS Block
Devices (RBD) directly on your {pve} cluster nodes, see
xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
hardware has plenty of CPU power and RAM, so running storage services
and VMs on the same node is possible.

To simplify management, we provide 'pveceph' - a tool to install and
manage {ceph} services on {pve} nodes.


Precondition
------------

To build a Proxmox Ceph Cluster there should be at least three (preferably)
identical servers for the setup.

A 10Gb network, exclusively used for Ceph, is recommended. A meshed
network setup is also an option if there are no 10Gb switches
available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] .

Check also the recommendations from
http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website].


Installation of Ceph Packages
-----------------------------

On each node run the installation script as follows:

[source,bash]
----
pveceph install
----

This sets up an `apt` package repository in
`/etc/apt/sources.list.d/ceph.list` and installs the required software.


Creating initial Ceph configuration
-----------------------------------

[thumbnail="gui-ceph-config.png"]

After installation of packages, you need to create an initial Ceph
configuration on just one node, based on your network (`10.10.10.0/24`
in the following example) dedicated for Ceph:

[source,bash]
----
pveceph init --network 10.10.10.0/24
----

This creates an initial config at `/etc/pve/ceph.conf`. That file is
automatically distributed to all {pve} nodes by using
xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link
from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run
Ceph commands without the need to specify a configuration file.


Creating Ceph Monitors
----------------------

[thumbnail="gui-ceph-monitor.png"]

On each node where a monitor is requested (three monitors are recommended)
create it by using the "Ceph" item in the GUI or run.


[source,bash]
----
pveceph createmon
----


Creating Ceph OSDs
------------------

[thumbnail="gui-ceph-osd-status.png"]

via GUI or via CLI as follows:

[source,bash]
----
pveceph createosd /dev/sd[X]
----

If you want to use a dedicated SSD journal disk:

NOTE: In order to use a dedicated journal disk (SSD), the disk needs
to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
partition table. You can create this with `gdisk /dev/sd(x)`. If there
is no GPT, you cannot select the disk as journal. Currently the
journal size is fixed to 5 GB.

[source,bash]
----
pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X]
----

Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD
journal disk.

[source,bash]
----
pveceph createosd /dev/sdf -journal_dev /dev/sdb
----

This partitions the disk (data and journal partition), creates
filesystems and starts the OSD, afterwards it is running and fully
functional. Please create at least 12 OSDs, distributed among your
nodes (4 OSDs on each node).

It should be noted that this command refuses to initialize disk when
it detects existing data. So if you want to overwrite a disk you
should remove existing data first. You can do that using:

[source,bash]
----
ceph-disk zap /dev/sd[X]
----

You can create OSDs containing both journal and data partitions or you
can place the journal on a dedicated SSD. Using a SSD journal disk is
highly recommended if you expect good performance.


Ceph Pools
----------

[thumbnail="gui-ceph-pools.png"]

The standard installation creates per default the pool 'rbd',
additional pools can be created via GUI.


Ceph Client
-----------

[thumbnail="gui-ceph-log.png"]

You can then configure {pve} to use such pools to store VM or
Container images. Simply use the GUI too add a new `RBD` storage (see
section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).

You also need to copy the keyring to a predefined location.

NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
`my-ceph-storage` in the following example:

[source,bash]
----
mkdir /etc/pve/priv/ceph
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
----


ifdef::manvolnum[]
include::pve-copyright.adoc[]
endif::manvolnum[]