]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[chapter_pveceph]] |
0840a663 | 2 | ifdef::manvolnum[] |
b2f242ab DM |
3 | pveceph(1) |
4 | ========== | |
404a158e | 5 | :pve-toplevel: |
0840a663 DM |
6 | |
7 | NAME | |
8 | ---- | |
9 | ||
21394e70 | 10 | pveceph - Manage Ceph Services on Proxmox VE Nodes |
0840a663 | 11 | |
49a5e11c | 12 | SYNOPSIS |
0840a663 DM |
13 | -------- |
14 | ||
15 | include::pveceph.1-synopsis.adoc[] | |
16 | ||
17 | DESCRIPTION | |
18 | ----------- | |
19 | endif::manvolnum[] | |
0840a663 | 20 | ifndef::manvolnum[] |
fe93f133 DM |
21 | Manage Ceph Services on Proxmox VE Nodes |
22 | ======================================== | |
49d3ad91 | 23 | :pve-toplevel: |
0840a663 DM |
24 | endif::manvolnum[] |
25 | ||
8997dd6e DM |
26 | [thumbnail="gui-ceph-status.png"] |
27 | ||
c994e4e5 DM |
28 | {pve} unifies your compute and storage systems, i.e. you can use the |
29 | same physical nodes within a cluster for both computing (processing | |
30 | VMs and containers) and replicated storage. The traditional silos of | |
31 | compute and storage resources can be wrapped up into a single | |
32 | hyper-converged appliance. Separate storage networks (SANs) and | |
33 | connections via network (NAS) disappear. With the integration of Ceph, | |
34 | an open source software-defined storage platform, {pve} has the | |
35 | ability to run and manage Ceph storage directly on the hypervisor | |
36 | nodes. | |
37 | ||
38 | Ceph is a distributed object store and file system designed to provide | |
39 | excellent performance, reliability and scalability. For smaller | |
40 | deployments, it is possible to install a Ceph server for RADOS Block | |
41 | Devices (RBD) directly on your {pve} cluster nodes, see | |
42 | xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent | |
43 | hardware has plenty of CPU power and RAM, so running storage services | |
44 | and VMs on the same node is possible. | |
21394e70 DM |
45 | |
46 | To simplify management, we provide 'pveceph' - a tool to install and | |
47 | manage {ceph} services on {pve} nodes. | |
48 | ||
49 | ||
50 | Precondition | |
51 | ------------ | |
52 | ||
c994e4e5 DM |
53 | To build a Proxmox Ceph Cluster there should be at least three (preferably) |
54 | identical servers for the setup. | |
21394e70 | 55 | |
470d4313 | 56 | A 10Gb network, exclusively used for Ceph, is recommended. A meshed |
c994e4e5 DM |
57 | network setup is also an option if there are no 10Gb switches |
58 | available, see {webwiki-url}Full_Mesh_Network_for_Ceph_Server[wiki] . | |
21394e70 DM |
59 | |
60 | Check also the recommendations from | |
19920184 | 61 | http://docs.ceph.com/docs/master/start/hardware-recommendations/[Ceph's website]. |
21394e70 DM |
62 | |
63 | ||
64 | Installation of Ceph Packages | |
65 | ----------------------------- | |
66 | ||
67 | On each node run the installation script as follows: | |
68 | ||
69 | [source,bash] | |
70 | ---- | |
19920184 | 71 | pveceph install |
21394e70 DM |
72 | ---- |
73 | ||
74 | This sets up an `apt` package repository in | |
75 | `/etc/apt/sources.list.d/ceph.list` and installs the required software. | |
76 | ||
77 | ||
78 | Creating initial Ceph configuration | |
79 | ----------------------------------- | |
80 | ||
8997dd6e DM |
81 | [thumbnail="gui-ceph-config.png"] |
82 | ||
21394e70 DM |
83 | After installation of packages, you need to create an initial Ceph |
84 | configuration on just one node, based on your network (`10.10.10.0/24` | |
85 | in the following example) dedicated for Ceph: | |
86 | ||
87 | [source,bash] | |
88 | ---- | |
89 | pveceph init --network 10.10.10.0/24 | |
90 | ---- | |
91 | ||
92 | This creates an initial config at `/etc/pve/ceph.conf`. That file is | |
c994e4e5 | 93 | automatically distributed to all {pve} nodes by using |
21394e70 DM |
94 | xref:chapter_pmxcfs[pmxcfs]. The command also creates a symbolic link |
95 | from `/etc/ceph/ceph.conf` pointing to that file. So you can simply run | |
96 | Ceph commands without the need to specify a configuration file. | |
97 | ||
98 | ||
99 | Creating Ceph Monitors | |
100 | ---------------------- | |
101 | ||
8997dd6e DM |
102 | [thumbnail="gui-ceph-monitor.png"] |
103 | ||
21394e70 DM |
104 | On each node where a monitor is requested (three monitors are recommended) |
105 | create it by using the "Ceph" item in the GUI or run. | |
106 | ||
107 | ||
108 | [source,bash] | |
109 | ---- | |
110 | pveceph createmon | |
111 | ---- | |
112 | ||
113 | ||
114 | Creating Ceph OSDs | |
115 | ------------------ | |
116 | ||
8997dd6e DM |
117 | [thumbnail="gui-ceph-osd-status.png"] |
118 | ||
21394e70 DM |
119 | via GUI or via CLI as follows: |
120 | ||
121 | [source,bash] | |
122 | ---- | |
123 | pveceph createosd /dev/sd[X] | |
124 | ---- | |
125 | ||
126 | If you want to use a dedicated SSD journal disk: | |
127 | ||
128 | NOTE: In order to use a dedicated journal disk (SSD), the disk needs | |
129 | to have a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT] | |
130 | partition table. You can create this with `gdisk /dev/sd(x)`. If there | |
131 | is no GPT, you cannot select the disk as journal. Currently the | |
132 | journal size is fixed to 5 GB. | |
133 | ||
134 | [source,bash] | |
135 | ---- | |
136 | pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X] | |
137 | ---- | |
138 | ||
139 | Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD | |
140 | journal disk. | |
141 | ||
142 | [source,bash] | |
143 | ---- | |
144 | pveceph createosd /dev/sdf -journal_dev /dev/sdb | |
145 | ---- | |
146 | ||
147 | This partitions the disk (data and journal partition), creates | |
148 | filesystems and starts the OSD, afterwards it is running and fully | |
149 | functional. Please create at least 12 OSDs, distributed among your | |
150 | nodes (4 OSDs on each node). | |
151 | ||
152 | It should be noted that this command refuses to initialize disk when | |
153 | it detects existing data. So if you want to overwrite a disk you | |
154 | should remove existing data first. You can do that using: | |
155 | ||
156 | [source,bash] | |
157 | ---- | |
158 | ceph-disk zap /dev/sd[X] | |
159 | ---- | |
160 | ||
161 | You can create OSDs containing both journal and data partitions or you | |
162 | can place the journal on a dedicated SSD. Using a SSD journal disk is | |
163 | highly recommended if you expect good performance. | |
164 | ||
165 | ||
166 | Ceph Pools | |
167 | ---------- | |
168 | ||
8997dd6e DM |
169 | [thumbnail="gui-ceph-pools.png"] |
170 | ||
21394e70 DM |
171 | The standard installation creates per default the pool 'rbd', |
172 | additional pools can be created via GUI. | |
173 | ||
174 | ||
175 | Ceph Client | |
176 | ----------- | |
177 | ||
8997dd6e DM |
178 | [thumbnail="gui-ceph-log.png"] |
179 | ||
21394e70 DM |
180 | You can then configure {pve} to use such pools to store VM or |
181 | Container images. Simply use the GUI too add a new `RBD` storage (see | |
182 | section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]). | |
183 | ||
184 | You also need to copy the keyring to a predefined location. | |
185 | ||
186 | NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is | |
187 | the expression after 'rbd:' in `/etc/pve/storage.cfg` which is | |
188 | `my-ceph-storage` in the following example: | |
189 | ||
190 | [source,bash] | |
191 | ---- | |
192 | mkdir /etc/pve/priv/ceph | |
193 | cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring | |
194 | ---- | |
0840a663 DM |
195 | |
196 | ||
197 | ifdef::manvolnum[] | |
198 | include::pve-copyright.adoc[] | |
199 | endif::manvolnum[] |