]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | ================================= |
2 | Network Configuration Reference | |
3 | ================================= | |
4 | ||
5 | Network configuration is critical for building a high performance :term:`Ceph | |
6 | Storage Cluster`. The Ceph Storage Cluster does not perform request routing or | |
7 | dispatching on behalf of the :term:`Ceph Client`. Instead, Ceph Clients make | |
8 | requests directly to Ceph OSD Daemons. Ceph OSD Daemons perform data replication | |
9 | on behalf of Ceph Clients, which means replication and other factors impose | |
10 | additional loads on Ceph Storage Cluster networks. | |
11 | ||
12 | Our Quick Start configurations provide a trivial `Ceph configuration file`_ that | |
13 | sets monitor IP addresses and daemon host names only. Unless you specify a | |
14 | cluster network, Ceph assumes a single "public" network. Ceph functions just | |
15 | fine with a public network only, but you may see significant performance | |
16 | improvement with a second "cluster" network in a large cluster. | |
17 | ||
18 | We recommend running a Ceph Storage Cluster with two networks: a public | |
19 | (front-side) network and a cluster (back-side) network. To support two networks, | |
20 | each :term:`Ceph Node` will need to have more than one NIC. See `Hardware | |
21 | Recommendations - Networks`_ for additional details. | |
22 | ||
23 | .. ditaa:: | |
24 | +-------------+ | |
25 | | Ceph Client | | |
26 | +----*--*-----+ | |
27 | | ^ | |
28 | Request | : Response | |
29 | v | | |
30 | /----------------------------------*--*-------------------------------------\ | |
31 | | Public Network | | |
32 | \---*--*------------*--*-------------*--*------------*--*------------*--*---/ | |
33 | ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | |
34 | | | | | | | | | | | | |
35 | | : | : | : | : | : | |
36 | v v v v v v v v v v | |
37 | +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+ | |
38 | | Ceph MON | | Ceph MDS | | Ceph OSD | | Ceph OSD | | Ceph OSD | | |
39 | +----------+ +----------+ +---*--*---+ +---*--*---+ +---*--*---+ | |
40 | ^ ^ ^ ^ ^ ^ | |
41 | The cluster network relieves | | | | | | | |
42 | OSD replication and heartbeat | : | : | : | |
43 | traffic from the public network. v v v v v v | |
44 | /------------------------------------*--*------------*--*------------*--*---\ | |
45 | | cCCC Cluster Network | | |
46 | \---------------------------------------------------------------------------/ | |
47 | ||
48 | ||
49 | There are several reasons to consider operating two separate networks: | |
50 | ||
51 | #. **Performance:** Ceph OSD Daemons handle data replication for the Ceph | |
52 | Clients. When Ceph OSD Daemons replicate data more than once, the network | |
53 | load between Ceph OSD Daemons easily dwarfs the network load between Ceph | |
54 | Clients and the Ceph Storage Cluster. This can introduce latency and | |
55 | create a performance problem. Recovery and rebalancing can | |
56 | also introduce significant latency on the public network. See | |
57 | `Scalability and High Availability`_ for additional details on how Ceph | |
58 | replicates data. See `Monitor / OSD Interaction`_ for details on heartbeat | |
59 | traffic. | |
60 | ||
61 | #. **Security**: While most people are generally civil, a very tiny segment of | |
62 | the population likes to engage in what's known as a Denial of Service (DoS) | |
63 | attack. When traffic between Ceph OSD Daemons gets disrupted, placement | |
64 | groups may no longer reflect an ``active + clean`` state, which may prevent | |
65 | users from reading and writing data. A great way to defeat this type of | |
66 | attack is to maintain a completely separate cluster network that doesn't | |
67 | connect directly to the internet. Also, consider using `Message Signatures`_ | |
68 | to defeat spoofing attacks. | |
69 | ||
70 | ||
71 | IP Tables | |
72 | ========= | |
73 | ||
74 | By default, daemons `bind`_ to ports within the ``6800:7300`` range. You may | |
75 | configure this range at your discretion. Before configuring your IP tables, | |
76 | check the default ``iptables`` configuration. | |
77 | ||
78 | sudo iptables -L | |
79 | ||
80 | Some Linux distributions include rules that reject all inbound requests | |
81 | except SSH from all network interfaces. For example:: | |
82 | ||
83 | REJECT all -- anywhere anywhere reject-with icmp-host-prohibited | |
84 | ||
85 | You will need to delete these rules on both your public and cluster networks | |
86 | initially, and replace them with appropriate rules when you are ready to | |
87 | harden the ports on your Ceph Nodes. | |
88 | ||
89 | ||
90 | Monitor IP Tables | |
91 | ----------------- | |
92 | ||
93 | Ceph Monitors listen on port ``6789`` by default. Additionally, Ceph Monitors | |
94 | always operate on the public network. When you add the rule using the example | |
95 | below, make sure you replace ``{iface}`` with the public network interface | |
96 | (e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP address of the | |
97 | public network and ``{netmask}`` with the netmask for the public network. :: | |
98 | ||
99 | sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT | |
100 | ||
101 | ||
102 | MDS IP Tables | |
103 | ------------- | |
104 | ||
105 | A :term:`Ceph Metadata Server` listens on the first available port on the public | |
106 | network beginning at port 6800. Note that this behavior is not deterministic, so | |
107 | if you are running more than one OSD or MDS on the same host, or if you restart | |
108 | the daemons within a short window of time, the daemons will bind to higher | |
109 | ports. You should open the entire 6800-7300 range by default. When you add the | |
110 | rule using the example below, make sure you replace ``{iface}`` with the public | |
111 | network interface (e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP | |
112 | address of the public network and ``{netmask}`` with the netmask of the public | |
113 | network. | |
114 | ||
115 | For example:: | |
116 | ||
117 | sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT | |
118 | ||
119 | ||
120 | OSD IP Tables | |
121 | ------------- | |
122 | ||
123 | By default, Ceph OSD Daemons `bind`_ to the first available ports on a Ceph Node | |
124 | beginning at port 6800. Note that this behavior is not deterministic, so if you | |
125 | are running more than one OSD or MDS on the same host, or if you restart the | |
126 | daemons within a short window of time, the daemons will bind to higher ports. | |
127 | Each Ceph OSD Daemon on a Ceph Node may use up to four ports: | |
128 | ||
129 | #. One for talking to clients and monitors. | |
130 | #. One for sending data to other OSDs. | |
131 | #. Two for heartbeating on each interface. | |
132 | ||
133 | .. ditaa:: | |
134 | /---------------\ | |
135 | | OSD | | |
136 | | +---+----------------+-----------+ | |
137 | | | Clients & Monitors | Heartbeat | | |
138 | | +---+----------------+-----------+ | |
139 | | | | |
140 | | +---+----------------+-----------+ | |
141 | | | Data Replication | Heartbeat | | |
142 | | +---+----------------+-----------+ | |
143 | | cCCC | | |
144 | \---------------/ | |
145 | ||
146 | When a daemon fails and restarts without letting go of the port, the restarted | |
147 | daemon will bind to a new port. You should open the entire 6800-7300 port range | |
148 | to handle this possibility. | |
149 | ||
150 | If you set up separate public and cluster networks, you must add rules for both | |
151 | the public network and the cluster network, because clients will connect using | |
152 | the public network and other Ceph OSD Daemons will connect using the cluster | |
153 | network. When you add the rule using the example below, make sure you replace | |
154 | ``{iface}`` with the network interface (e.g., ``eth0``, ``eth1``, etc.), | |
155 | ``{ip-address}`` with the IP address and ``{netmask}`` with the netmask of the | |
156 | public or cluster network. For example:: | |
157 | ||
158 | sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT | |
159 | ||
160 | .. tip:: If you run Ceph Metadata Servers on the same Ceph Node as the | |
161 | Ceph OSD Daemons, you can consolidate the public network configuration step. | |
162 | ||
163 | ||
164 | Ceph Networks | |
165 | ============= | |
166 | ||
167 | To configure Ceph networks, you must add a network configuration to the | |
168 | ``[global]`` section of the configuration file. Our 5-minute Quick Start | |
169 | provides a trivial `Ceph configuration file`_ that assumes one public network | |
170 | with client and server on the same network and subnet. Ceph functions just fine | |
171 | with a public network only. However, Ceph allows you to establish much more | |
172 | specific criteria, including multiple IP network and subnet masks for your | |
173 | public network. You can also establish a separate cluster network to handle OSD | |
174 | heartbeat, object replication and recovery traffic. Don't confuse the IP | |
175 | addresses you set in your configuration with the public-facing IP addresses | |
176 | network clients may use to access your service. Typical internal IP networks are | |
177 | often ``192.168.0.0`` or ``10.0.0.0``. | |
178 | ||
179 | .. tip:: If you specify more than one IP address and subnet mask for | |
180 | either the public or the cluster network, the subnets within the network | |
181 | must be capable of routing to each other. Additionally, make sure you | |
182 | include each IP address/subnet in your IP tables and open ports for them | |
183 | as necessary. | |
184 | ||
185 | .. note:: Ceph uses `CIDR`_ notation for subnets (e.g., ``10.0.0.0/24``). | |
186 | ||
c07f9fc5 | 187 | When you have configured your networks, you may restart your cluster or restart |
7c673cae FG |
188 | each daemon. Ceph daemons bind dynamically, so you do not have to restart the |
189 | entire cluster at once if you change your network configuration. | |
190 | ||
191 | ||
192 | Public Network | |
193 | -------------- | |
194 | ||
195 | To configure a public network, add the following option to the ``[global]`` | |
196 | section of your Ceph configuration file. | |
197 | ||
198 | .. code-block:: ini | |
199 | ||
200 | [global] | |
201 | ... | |
202 | public network = {public-network/netmask} | |
203 | ||
204 | ||
205 | Cluster Network | |
206 | --------------- | |
207 | ||
208 | If you declare a cluster network, OSDs will route heartbeat, object replication | |
209 | and recovery traffic over the cluster network. This may improve performance | |
210 | compared to using a single network. To configure a cluster network, add the | |
211 | following option to the ``[global]`` section of your Ceph configuration file. | |
212 | ||
213 | .. code-block:: ini | |
214 | ||
215 | [global] | |
216 | ... | |
217 | cluster network = {cluster-network/netmask} | |
218 | ||
219 | We prefer that the cluster network is **NOT** reachable from the public network | |
220 | or the Internet for added security. | |
221 | ||
222 | ||
223 | Ceph Daemons | |
224 | ============ | |
225 | ||
226 | Ceph has one network configuration requirement that applies to all daemons: the | |
227 | Ceph configuration file **MUST** specify the ``host`` for each daemon. Ceph also | |
228 | requires that a Ceph configuration file specify the monitor IP address and its | |
229 | port. | |
230 | ||
231 | .. important:: Some deployment tools (e.g., ``ceph-deploy``, Chef) may create a | |
232 | configuration file for you. **DO NOT** set these values if the deployment | |
233 | tool does it for you. | |
234 | ||
235 | .. tip:: The ``host`` setting is the short name of the host (i.e., not | |
236 | an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on | |
237 | the command line to retrieve the name of the host. | |
238 | ||
239 | ||
240 | .. code-block:: ini | |
241 | ||
242 | [mon.a] | |
243 | ||
244 | host = {hostname} | |
245 | mon addr = {ip-address}:6789 | |
246 | ||
247 | [osd.0] | |
248 | host = {hostname} | |
249 | ||
250 | ||
251 | You do not have to set the host IP address for a daemon. If you have a static IP | |
252 | configuration and both public and cluster networks running, the Ceph | |
253 | configuration file may specify the IP address of the host for each daemon. To | |
254 | set a static IP address for a daemon, the following option(s) should appear in | |
255 | the daemon instance sections of your ``ceph.conf`` file. | |
256 | ||
257 | .. code-block:: ini | |
258 | ||
259 | [osd.0] | |
260 | public addr = {host-public-ip-address} | |
261 | cluster addr = {host-cluster-ip-address} | |
262 | ||
263 | ||
264 | .. topic:: One NIC OSD in a Two Network Cluster | |
265 | ||
266 | Generally, we do not recommend deploying an OSD host with a single NIC in a | |
267 | cluster with two networks. However, you may accomplish this by forcing the | |
268 | OSD host to operate on the public network by adding a ``public addr`` entry | |
269 | to the ``[osd.n]`` section of the Ceph configuration file, where ``n`` | |
270 | refers to the number of the OSD with one NIC. Additionally, the public | |
271 | network and cluster network must be able to route traffic to each other, | |
272 | which we don't recommend for security reasons. | |
273 | ||
274 | ||
275 | Network Config Settings | |
276 | ======================= | |
277 | ||
278 | Network configuration settings are not required. Ceph assumes a public network | |
279 | with all hosts operating on it unless you specifically configure a cluster | |
280 | network. | |
281 | ||
282 | ||
283 | Public Network | |
284 | -------------- | |
285 | ||
286 | The public network configuration allows you specifically define IP addresses | |
287 | and subnets for the public network. You may specifically assign static IP | |
288 | addresses or override ``public network`` settings using the ``public addr`` | |
289 | setting for a specific daemon. | |
290 | ||
291 | ``public network`` | |
292 | ||
293 | :Description: The IP address and netmask of the public (front-side) network | |
294 | (e.g., ``192.168.0.0/24``). Set in ``[global]``. You may specify | |
295 | comma-delimited subnets. | |
296 | ||
297 | :Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]`` | |
298 | :Required: No | |
299 | :Default: N/A | |
300 | ||
301 | ||
302 | ``public addr`` | |
303 | ||
304 | :Description: The IP address for the public (front-side) network. | |
305 | Set for each daemon. | |
306 | ||
307 | :Type: IP Address | |
308 | :Required: No | |
309 | :Default: N/A | |
310 | ||
311 | ||
312 | ||
313 | Cluster Network | |
314 | --------------- | |
315 | ||
316 | The cluster network configuration allows you to declare a cluster network, and | |
317 | specifically define IP addresses and subnets for the cluster network. You may | |
318 | specifically assign static IP addresses or override ``cluster network`` | |
319 | settings using the ``cluster addr`` setting for specific OSD daemons. | |
320 | ||
321 | ||
322 | ``cluster network`` | |
323 | ||
324 | :Description: The IP address and netmask of the cluster (back-side) network | |
325 | (e.g., ``10.0.0.0/24``). Set in ``[global]``. You may specify | |
326 | comma-delimited subnets. | |
327 | ||
328 | :Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]`` | |
329 | :Required: No | |
330 | :Default: N/A | |
331 | ||
332 | ||
333 | ``cluster addr`` | |
334 | ||
335 | :Description: The IP address for the cluster (back-side) network. | |
336 | Set for each daemon. | |
337 | ||
338 | :Type: Address | |
339 | :Required: No | |
340 | :Default: N/A | |
341 | ||
342 | ||
343 | Bind | |
344 | ---- | |
345 | ||
346 | Bind settings set the default port ranges Ceph OSD and MDS daemons use. The | |
347 | default range is ``6800:7300``. Ensure that your `IP Tables`_ configuration | |
348 | allows you to use the configured port range. | |
349 | ||
350 | You may also enable Ceph daemons to bind to IPv6 addresses instead of IPv4 | |
351 | addresses. | |
352 | ||
353 | ||
354 | ``ms bind port min`` | |
355 | ||
356 | :Description: The minimum port number to which an OSD or MDS daemon will bind. | |
357 | :Type: 32-bit Integer | |
358 | :Default: ``6800`` | |
359 | :Required: No | |
360 | ||
361 | ||
362 | ``ms bind port max`` | |
363 | ||
364 | :Description: The maximum port number to which an OSD or MDS daemon will bind. | |
365 | :Type: 32-bit Integer | |
366 | :Default: ``7300`` | |
367 | :Required: No. | |
368 | ||
369 | ||
370 | ``ms bind ipv6`` | |
371 | ||
372 | :Description: Enables Ceph daemons to bind to IPv6 addresses. Currently the | |
c07f9fc5 | 373 | messenger *either* uses IPv4 or IPv6, but it cannot do both. |
7c673cae FG |
374 | :Type: Boolean |
375 | :Default: ``false`` | |
376 | :Required: No | |
377 | ||
224ce89b WB |
378 | ``public bind addr`` |
379 | ||
380 | :Description: In some dynamic deployments the Ceph MON daemon might bind | |
381 | to an IP address locally that is different from the ``public addr`` | |
382 | advertised to other peers in the network. The environment must ensure | |
383 | that routing rules are set correclty. If ``public bind addr`` is set | |
384 | the Ceph MON daemon will bind to it locally and use ``public addr`` | |
385 | in the monmaps to advertise its address to peers. This behavior is limited | |
386 | to the MON daemon. | |
387 | ||
388 | :Type: IP Address | |
389 | :Required: No | |
390 | :Default: N/A | |
391 | ||
7c673cae FG |
392 | |
393 | ||
394 | Hosts | |
395 | ----- | |
396 | ||
397 | Ceph expects at least one monitor declared in the Ceph configuration file, with | |
398 | a ``mon addr`` setting under each declared monitor. Ceph expects a ``host`` | |
399 | setting under each declared monitor, metadata server and OSD in the Ceph | |
224ce89b WB |
400 | configuration file. Optionally, a monitor can be assigned with a priority, and |
401 | the clients will always connect to the monitor with lower value of priority if | |
402 | specified. | |
7c673cae FG |
403 | |
404 | ||
405 | ``mon addr`` | |
406 | ||
407 | :Description: A list of ``{hostname}:{port}`` entries that clients can use to | |
408 | connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]`` | |
409 | sections. | |
410 | ||
411 | :Type: String | |
412 | :Required: No | |
413 | :Default: N/A | |
414 | ||
224ce89b WB |
415 | ``mon priority`` |
416 | ||
417 | :Description: The priority of the declared monitor, the lower value the more | |
418 | prefered when a client selects a monitor when trying to connect | |
419 | to the cluster. | |
420 | ||
421 | :Type: Unsigned 16-bit Integer | |
422 | :Required: No | |
423 | :Default: 0 | |
7c673cae FG |
424 | |
425 | ``host`` | |
426 | ||
427 | :Description: The hostname. Use this setting for specific daemon instances | |
428 | (e.g., ``[osd.0]``). | |
429 | ||
430 | :Type: String | |
431 | :Required: Yes, for daemon instances. | |
432 | :Default: ``localhost`` | |
433 | ||
434 | .. tip:: Do not use ``localhost``. To get your host name, execute | |
435 | ``hostname -s`` on your command line and use the name of your host | |
436 | (to the first period, not the fully-qualified domain name). | |
437 | ||
438 | .. important:: You should not specify any value for ``host`` when using a third | |
439 | party deployment system that retrieves the host name for you. | |
440 | ||
441 | ||
442 | ||
443 | TCP | |
444 | --- | |
445 | ||
446 | Ceph disables TCP buffering by default. | |
447 | ||
448 | ||
449 | ``ms tcp nodelay`` | |
450 | ||
451 | :Description: Ceph enables ``ms tcp nodelay`` so that each request is sent | |
452 | immediately (no buffering). Disabling `Nagle's algorithm`_ | |
453 | increases network traffic, which can introduce latency. If you | |
454 | experience large numbers of small packets, you may try | |
455 | disabling ``ms tcp nodelay``. | |
456 | ||
457 | :Type: Boolean | |
458 | :Required: No | |
459 | :Default: ``true`` | |
460 | ||
461 | ||
462 | ||
463 | ``ms tcp rcvbuf`` | |
464 | ||
465 | :Description: The size of the socket buffer on the receiving end of a network | |
466 | connection. Disable by default. | |
467 | ||
468 | :Type: 32-bit Integer | |
469 | :Required: No | |
470 | :Default: ``0`` | |
471 | ||
472 | ||
473 | ||
474 | ``ms tcp read timeout`` | |
475 | ||
476 | :Description: If a client or daemon makes a request to another Ceph daemon and | |
477 | does not drop an unused connection, the ``ms tcp read timeout`` | |
478 | defines the connection as idle after the specified number | |
479 | of seconds. | |
480 | ||
481 | :Type: Unsigned 64-bit Integer | |
482 | :Required: No | |
483 | :Default: ``900`` 15 minutes. | |
484 | ||
485 | ||
486 | ||
487 | .. _Scalability and High Availability: ../../../architecture#scalability-and-high-availability | |
488 | .. _Hardware Recommendations - Networks: ../../../start/hardware-recommendations#networks | |
489 | .. _Ceph configuration file: ../../../start/quick-ceph-deploy/#create-a-cluster | |
490 | .. _hardware recommendations: ../../../start/hardware-recommendations | |
491 | .. _Monitor / OSD Interaction: ../mon-osd-interaction | |
492 | .. _Message Signatures: ../auth-config-ref#signatures | |
493 | .. _CIDR: http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing | |
494 | .. _Nagle's Algorithm: http://en.wikipedia.org/wiki/Nagle's_algorithm |