]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[chapter_pveceph]] |
0840a663 | 2 | ifdef::manvolnum[] |
b2f242ab DM |
3 | pveceph(1) |
4 | ========== | |
404a158e | 5 | :pve-toplevel: |
0840a663 DM |
6 | |
7 | NAME | |
8 | ---- | |
9 | ||
21394e70 | 10 | pveceph - Manage Ceph Services on Proxmox VE Nodes |
0840a663 | 11 | |
49a5e11c | 12 | SYNOPSIS |
0840a663 DM |
13 | -------- |
14 | ||
15 | include::pveceph.1-synopsis.adoc[] | |
16 | ||
17 | DESCRIPTION | |
18 | ----------- | |
19 | endif::manvolnum[] | |
0840a663 | 20 | ifndef::manvolnum[] |
fe93f133 DM |
21 | Manage Ceph Services on Proxmox VE Nodes |
22 | ======================================== | |
49d3ad91 | 23 | :pve-toplevel: |
0840a663 DM |
24 | endif::manvolnum[] |
25 | ||
8997dd6e DM |
26 | [thumbnail="gui-ceph-status.png"] |
27 | ||
c994e4e5 DM |
28 | {pve} unifies your compute and storage systems, i.e. you can use the |
29 | same physical nodes within a cluster for both computing (processing | |
30 | VMs and containers) and replicated storage. The traditional silos of | |
31 | compute and storage resources can be wrapped up into a single | |
32 | hyper-converged appliance. Separate storage networks (SANs) and | |
33 | connections via network (NAS) disappear. With the integration of Ceph, | |
34 | an open source software-defined storage platform, {pve} has the | |
35 | ability to run and manage Ceph storage directly on the hypervisor | |
36 | nodes. | |
37 | ||
38 | Ceph is a distributed object store and file system designed to provide | |
39 | excellent performance, reliability and scalability. For smaller | |
40 | deployments, it is possible to install a Ceph server for RADOS Block | |
41 | Devices (RBD) directly on your {pve} cluster nodes, see | |
42 | xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent | |
43 | hardware has plenty of CPU power and RAM, so running storage services | |
44 | and VMs on the same node is possible. | |
21394e70 DM |
45 | |
46 | To simplify management, we provide 'pveceph' - a tool to install and | |
47 | manage {ceph} services on {pve} nodes. | |
48 | ||
49 | ||
50 | Precondition | |
51 | ------------ | |
52 | ||
c994e4e5 DM |
53 | To build a Proxmox Ceph Cluster there should be at least three (preferably) |
54 | identical servers for the setup. | |
21394e70 | 55 | |
470d4313 | 56 | A 10Gb network, exclusively used for Ceph, is recommended. A meshed |
c994e4e5 DM |
57 | network setup is also an option if there are no 10Gb switches |
58 | available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] . | |
21394e70 DM |
59 | |
60 | Check also the recommendations from | |
19920184 | 61 | http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website]. |
21394e70 DM |
62 | |
63 | ||
64 | Installation of Ceph Packages | |
65 | ----------------------------- | |
66 | ||
67 | On each node run the installation script as follows: | |
68 | ||
69 | [source,bash] | |
70 | ---- | |
19920184 | 71 | pveceph install |
21394e70 DM |
72 | ---- |
73 | ||
74 | This sets up an `apt` package repository in | |
75 | `/etc/apt/sources.list.d/ceph.list` and installs the required software. | |
76 | ||
77 | ||
78 | Creating initial Ceph configuration | |
79 | ----------------------------------- | |
80 | ||
8997dd6e DM |
81 | [thumbnail="gui-ceph-config.png"] |
82 | ||
21394e70 DM |
83 | After installation of packages, you need to create an initial Ceph |
84 | configuration on just one node, based on your network (`10.10.10.0/24` | |
85 | in the following example) dedicated for Ceph: | |
86 | ||
87 | [source,bash] | |
88 | ---- | |
89 | pveceph init --network 10.10.10.0/24 | |
90 | ---- | |
91 | ||
92 | This creates an initial config at `/etc/pve/ceph.conf`. That file is | |
c994e4e5 | 93 | automatically distributed to all {pve} nodes by using |
21394e70 DM |
94 | xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link |
95 | from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run | |
96 | Ceph commands without the need to specify a configuration file. | |
97 | ||
98 | ||
d9a27ee1 | 99 | [[pve_ceph_monitors]] |
21394e70 DM |
100 | Creating Ceph Monitors |
101 | ---------------------- | |
102 | ||
8997dd6e DM |
103 | [thumbnail="gui-ceph-monitor.png"] |
104 | ||
21394e70 DM |
105 | On each node where a monitor is requested (three monitors are recommended) |
106 | create it by using the "Ceph" item in the GUI or run. | |
107 | ||
108 | ||
109 | [source,bash] | |
110 | ---- | |
111 | pveceph createmon | |
112 | ---- | |
113 | ||
114 | ||
d9a27ee1 | 115 | [[pve_ceph_osds]] |
21394e70 DM |
116 | Creating Ceph OSDs |
117 | ------------------ | |
118 | ||
8997dd6e DM |
119 | [thumbnail="gui-ceph-osd-status.png"] |
120 | ||
21394e70 DM |
121 | via GUI or via CLI as follows: |
122 | ||
123 | [source,bash] | |
124 | ---- | |
125 | pveceph createosd /dev/sd[X] | |
126 | ---- | |
127 | ||
128 | If you want to use a dedicated SSD journal disk: | |
129 | ||
130 | NOTE: In order to use a dedicated journal disk (SSD), the disk needs | |
131 | to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT] | |
132 | partition table. You can create this with `gdisk /dev/sd(x)`. If there | |
133 | is no GPT, you cannot select the disk as journal. Currently the | |
134 | journal size is fixed to 5 GB. | |
135 | ||
136 | [source,bash] | |
137 | ---- | |
138 | pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X] | |
139 | ---- | |
140 | ||
141 | Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD | |
142 | journal disk. | |
143 | ||
144 | [source,bash] | |
145 | ---- | |
146 | pveceph createosd /dev/sdf -journal_dev /dev/sdb | |
147 | ---- | |
148 | ||
149 | This partitions the disk (data and journal partition), creates | |
150 | filesystems and starts the OSD, afterwards it is running and fully | |
151 | functional. Please create at least 12 OSDs, distributed among your | |
152 | nodes (4 OSDs on each node). | |
153 | ||
154 | It should be noted that this command refuses to initialize disk when | |
155 | it detects existing data. So if you want to overwrite a disk you | |
156 | should remove existing data first. You can do that using: | |
157 | ||
158 | [source,bash] | |
159 | ---- | |
160 | ceph-disk zap /dev/sd[X] | |
161 | ---- | |
162 | ||
163 | You can create OSDs containing both journal and data partitions or you | |
164 | can place the journal on a dedicated SSD. Using a SSD journal disk is | |
165 | highly recommended if you expect good performance. | |
166 | ||
167 | ||
d9a27ee1 | 168 | [[pve_ceph_pools]] |
21394e70 DM |
169 | Ceph Pools |
170 | ---------- | |
171 | ||
8997dd6e DM |
172 | [thumbnail="gui-ceph-pools.png"] |
173 | ||
21394e70 DM |
174 | The standard installation creates per default the pool 'rbd', |
175 | additional pools can be created via GUI. | |
176 | ||
177 | ||
178 | Ceph Client | |
179 | ----------- | |
180 | ||
8997dd6e DM |
181 | [thumbnail="gui-ceph-log.png"] |
182 | ||
21394e70 DM |
183 | You can then configure {pve} to use such pools to store VM or |
184 | Container images. Simply use the GUI too add a new `RBD` storage (see | |
185 | section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]). | |
186 | ||
187 | You also need to copy the keyring to a predefined location. | |
188 | ||
189 | NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is | |
190 | the expression after 'rbd:' in `/etc/pve/storage.cfg` which is | |
191 | `my-ceph-storage` in the following example: | |
192 | ||
193 | [source,bash] | |
194 | ---- | |
195 | mkdir /etc/pve/priv/ceph | |
196 | cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring | |
197 | ---- | |
0840a663 DM |
198 | |
199 | ||
200 | ifdef::manvolnum[] | |
201 | include::pve-copyright.adoc[] | |
202 | endif::manvolnum[] |