]> git.proxmox.com Git - ceph.git/blob - ceph/doc/rados/configuration/network-config-ref.rst
update sources to v12.1.1
[ceph.git] / ceph / doc / rados / configuration / network-config-ref.rst
1 =================================
2 Network Configuration Reference
3 =================================
4
5 Network configuration is critical for building a high performance :term:`Ceph
6 Storage Cluster`. The Ceph Storage Cluster does not perform request routing or
7 dispatching on behalf of the :term:`Ceph Client`. Instead, Ceph Clients make
8 requests directly to Ceph OSD Daemons. Ceph OSD Daemons perform data replication
9 on behalf of Ceph Clients, which means replication and other factors impose
10 additional loads on Ceph Storage Cluster networks.
11
12 Our Quick Start configurations provide a trivial `Ceph configuration file`_ that
13 sets monitor IP addresses and daemon host names only. Unless you specify a
14 cluster network, Ceph assumes a single "public" network. Ceph functions just
15 fine with a public network only, but you may see significant performance
16 improvement with a second "cluster" network in a large cluster.
17
18 We recommend running a Ceph Storage Cluster with two networks: a public
19 (front-side) network and a cluster (back-side) network. To support two networks,
20 each :term:`Ceph Node` will need to have more than one NIC. See `Hardware
21 Recommendations - Networks`_ for additional details.
22
23 .. ditaa::
24 +-------------+
25 | Ceph Client |
26 +----*--*-----+
27 | ^
28 Request | : Response
29 v |
30 /----------------------------------*--*-------------------------------------\
31 | Public Network |
32 \---*--*------------*--*-------------*--*------------*--*------------*--*---/
33 ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
34 | | | | | | | | | |
35 | : | : | : | : | :
36 v v v v v v v v v v
37 +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+
38 | Ceph MON | | Ceph MDS | | Ceph OSD | | Ceph OSD | | Ceph OSD |
39 +----------+ +----------+ +---*--*---+ +---*--*---+ +---*--*---+
40 ^ ^ ^ ^ ^ ^
41 The cluster network relieves | | | | | |
42 OSD replication and heartbeat | : | : | :
43 traffic from the public network. v v v v v v
44 /------------------------------------*--*------------*--*------------*--*---\
45 | cCCC Cluster Network |
46 \---------------------------------------------------------------------------/
47
48
49 There are several reasons to consider operating two separate networks:
50
51 #. **Performance:** Ceph OSD Daemons handle data replication for the Ceph
52 Clients. When Ceph OSD Daemons replicate data more than once, the network
53 load between Ceph OSD Daemons easily dwarfs the network load between Ceph
54 Clients and the Ceph Storage Cluster. This can introduce latency and
55 create a performance problem. Recovery and rebalancing can
56 also introduce significant latency on the public network. See
57 `Scalability and High Availability`_ for additional details on how Ceph
58 replicates data. See `Monitor / OSD Interaction`_ for details on heartbeat
59 traffic.
60
61 #. **Security**: While most people are generally civil, a very tiny segment of
62 the population likes to engage in what's known as a Denial of Service (DoS)
63 attack. When traffic between Ceph OSD Daemons gets disrupted, placement
64 groups may no longer reflect an ``active + clean`` state, which may prevent
65 users from reading and writing data. A great way to defeat this type of
66 attack is to maintain a completely separate cluster network that doesn't
67 connect directly to the internet. Also, consider using `Message Signatures`_
68 to defeat spoofing attacks.
69
70
71 IP Tables
72 =========
73
74 By default, daemons `bind`_ to ports within the ``6800:7300`` range. You may
75 configure this range at your discretion. Before configuring your IP tables,
76 check the default ``iptables`` configuration.
77
78 sudo iptables -L
79
80 Some Linux distributions include rules that reject all inbound requests
81 except SSH from all network interfaces. For example::
82
83 REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
84
85 You will need to delete these rules on both your public and cluster networks
86 initially, and replace them with appropriate rules when you are ready to
87 harden the ports on your Ceph Nodes.
88
89
90 Monitor IP Tables
91 -----------------
92
93 Ceph Monitors listen on port ``6789`` by default. Additionally, Ceph Monitors
94 always operate on the public network. When you add the rule using the example
95 below, make sure you replace ``{iface}`` with the public network interface
96 (e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP address of the
97 public network and ``{netmask}`` with the netmask for the public network. ::
98
99 sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT
100
101
102 MDS IP Tables
103 -------------
104
105 A :term:`Ceph Metadata Server` listens on the first available port on the public
106 network beginning at port 6800. Note that this behavior is not deterministic, so
107 if you are running more than one OSD or MDS on the same host, or if you restart
108 the daemons within a short window of time, the daemons will bind to higher
109 ports. You should open the entire 6800-7300 range by default. When you add the
110 rule using the example below, make sure you replace ``{iface}`` with the public
111 network interface (e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP
112 address of the public network and ``{netmask}`` with the netmask of the public
113 network.
114
115 For example::
116
117 sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT
118
119
120 OSD IP Tables
121 -------------
122
123 By default, Ceph OSD Daemons `bind`_ to the first available ports on a Ceph Node
124 beginning at port 6800. Note that this behavior is not deterministic, so if you
125 are running more than one OSD or MDS on the same host, or if you restart the
126 daemons within a short window of time, the daemons will bind to higher ports.
127 Each Ceph OSD Daemon on a Ceph Node may use up to four ports:
128
129 #. One for talking to clients and monitors.
130 #. One for sending data to other OSDs.
131 #. Two for heartbeating on each interface.
132
133 .. ditaa::
134 /---------------\
135 | OSD |
136 | +---+----------------+-----------+
137 | | Clients & Monitors | Heartbeat |
138 | +---+----------------+-----------+
139 | |
140 | +---+----------------+-----------+
141 | | Data Replication | Heartbeat |
142 | +---+----------------+-----------+
143 | cCCC |
144 \---------------/
145
146 When a daemon fails and restarts without letting go of the port, the restarted
147 daemon will bind to a new port. You should open the entire 6800-7300 port range
148 to handle this possibility.
149
150 If you set up separate public and cluster networks, you must add rules for both
151 the public network and the cluster network, because clients will connect using
152 the public network and other Ceph OSD Daemons will connect using the cluster
153 network. When you add the rule using the example below, make sure you replace
154 ``{iface}`` with the network interface (e.g., ``eth0``, ``eth1``, etc.),
155 ``{ip-address}`` with the IP address and ``{netmask}`` with the netmask of the
156 public or cluster network. For example::
157
158 sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT
159
160 .. tip:: If you run Ceph Metadata Servers on the same Ceph Node as the
161 Ceph OSD Daemons, you can consolidate the public network configuration step.
162
163
164 Ceph Networks
165 =============
166
167 To configure Ceph networks, you must add a network configuration to the
168 ``[global]`` section of the configuration file. Our 5-minute Quick Start
169 provides a trivial `Ceph configuration file`_ that assumes one public network
170 with client and server on the same network and subnet. Ceph functions just fine
171 with a public network only. However, Ceph allows you to establish much more
172 specific criteria, including multiple IP network and subnet masks for your
173 public network. You can also establish a separate cluster network to handle OSD
174 heartbeat, object replication and recovery traffic. Don't confuse the IP
175 addresses you set in your configuration with the public-facing IP addresses
176 network clients may use to access your service. Typical internal IP networks are
177 often ``192.168.0.0`` or ``10.0.0.0``.
178
179 .. tip:: If you specify more than one IP address and subnet mask for
180 either the public or the cluster network, the subnets within the network
181 must be capable of routing to each other. Additionally, make sure you
182 include each IP address/subnet in your IP tables and open ports for them
183 as necessary.
184
185 .. note:: Ceph uses `CIDR`_ notation for subnets (e.g., ``10.0.0.0/24``).
186
187 When you've configured your networks, you may restart your cluster or restart
188 each daemon. Ceph daemons bind dynamically, so you do not have to restart the
189 entire cluster at once if you change your network configuration.
190
191
192 Public Network
193 --------------
194
195 To configure a public network, add the following option to the ``[global]``
196 section of your Ceph configuration file.
197
198 .. code-block:: ini
199
200 [global]
201 ...
202 public network = {public-network/netmask}
203
204
205 Cluster Network
206 ---------------
207
208 If you declare a cluster network, OSDs will route heartbeat, object replication
209 and recovery traffic over the cluster network. This may improve performance
210 compared to using a single network. To configure a cluster network, add the
211 following option to the ``[global]`` section of your Ceph configuration file.
212
213 .. code-block:: ini
214
215 [global]
216 ...
217 cluster network = {cluster-network/netmask}
218
219 We prefer that the cluster network is **NOT** reachable from the public network
220 or the Internet for added security.
221
222
223 Ceph Daemons
224 ============
225
226 Ceph has one network configuration requirement that applies to all daemons: the
227 Ceph configuration file **MUST** specify the ``host`` for each daemon. Ceph also
228 requires that a Ceph configuration file specify the monitor IP address and its
229 port.
230
231 .. important:: Some deployment tools (e.g., ``ceph-deploy``, Chef) may create a
232 configuration file for you. **DO NOT** set these values if the deployment
233 tool does it for you.
234
235 .. tip:: The ``host`` setting is the short name of the host (i.e., not
236 an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on
237 the command line to retrieve the name of the host.
238
239
240 .. code-block:: ini
241
242 [mon.a]
243
244 host = {hostname}
245 mon addr = {ip-address}:6789
246
247 [osd.0]
248 host = {hostname}
249
250
251 You do not have to set the host IP address for a daemon. If you have a static IP
252 configuration and both public and cluster networks running, the Ceph
253 configuration file may specify the IP address of the host for each daemon. To
254 set a static IP address for a daemon, the following option(s) should appear in
255 the daemon instance sections of your ``ceph.conf`` file.
256
257 .. code-block:: ini
258
259 [osd.0]
260 public addr = {host-public-ip-address}
261 cluster addr = {host-cluster-ip-address}
262
263
264 .. topic:: One NIC OSD in a Two Network Cluster
265
266 Generally, we do not recommend deploying an OSD host with a single NIC in a
267 cluster with two networks. However, you may accomplish this by forcing the
268 OSD host to operate on the public network by adding a ``public addr`` entry
269 to the ``[osd.n]`` section of the Ceph configuration file, where ``n``
270 refers to the number of the OSD with one NIC. Additionally, the public
271 network and cluster network must be able to route traffic to each other,
272 which we don't recommend for security reasons.
273
274
275 Network Config Settings
276 =======================
277
278 Network configuration settings are not required. Ceph assumes a public network
279 with all hosts operating on it unless you specifically configure a cluster
280 network.
281
282
283 Public Network
284 --------------
285
286 The public network configuration allows you specifically define IP addresses
287 and subnets for the public network. You may specifically assign static IP
288 addresses or override ``public network`` settings using the ``public addr``
289 setting for a specific daemon.
290
291 ``public network``
292
293 :Description: The IP address and netmask of the public (front-side) network
294 (e.g., ``192.168.0.0/24``). Set in ``[global]``. You may specify
295 comma-delimited subnets.
296
297 :Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]``
298 :Required: No
299 :Default: N/A
300
301
302 ``public addr``
303
304 :Description: The IP address for the public (front-side) network.
305 Set for each daemon.
306
307 :Type: IP Address
308 :Required: No
309 :Default: N/A
310
311
312
313 Cluster Network
314 ---------------
315
316 The cluster network configuration allows you to declare a cluster network, and
317 specifically define IP addresses and subnets for the cluster network. You may
318 specifically assign static IP addresses or override ``cluster network``
319 settings using the ``cluster addr`` setting for specific OSD daemons.
320
321
322 ``cluster network``
323
324 :Description: The IP address and netmask of the cluster (back-side) network
325 (e.g., ``10.0.0.0/24``). Set in ``[global]``. You may specify
326 comma-delimited subnets.
327
328 :Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]``
329 :Required: No
330 :Default: N/A
331
332
333 ``cluster addr``
334
335 :Description: The IP address for the cluster (back-side) network.
336 Set for each daemon.
337
338 :Type: Address
339 :Required: No
340 :Default: N/A
341
342
343 Bind
344 ----
345
346 Bind settings set the default port ranges Ceph OSD and MDS daemons use. The
347 default range is ``6800:7300``. Ensure that your `IP Tables`_ configuration
348 allows you to use the configured port range.
349
350 You may also enable Ceph daemons to bind to IPv6 addresses instead of IPv4
351 addresses.
352
353
354 ``ms bind port min``
355
356 :Description: The minimum port number to which an OSD or MDS daemon will bind.
357 :Type: 32-bit Integer
358 :Default: ``6800``
359 :Required: No
360
361
362 ``ms bind port max``
363
364 :Description: The maximum port number to which an OSD or MDS daemon will bind.
365 :Type: 32-bit Integer
366 :Default: ``7300``
367 :Required: No.
368
369
370 ``ms bind ipv6``
371
372 :Description: Enables Ceph daemons to bind to IPv6 addresses. Currently the
373 messenger *either* uses IPv4 or IPv6, but it can't do both.
374 :Type: Boolean
375 :Default: ``false``
376 :Required: No
377
378 ``public bind addr``
379
380 :Description: In some dynamic deployments the Ceph MON daemon might bind
381 to an IP address locally that is different from the ``public addr``
382 advertised to other peers in the network. The environment must ensure
383 that routing rules are set correclty. If ``public bind addr`` is set
384 the Ceph MON daemon will bind to it locally and use ``public addr``
385 in the monmaps to advertise its address to peers. This behavior is limited
386 to the MON daemon.
387
388 :Type: IP Address
389 :Required: No
390 :Default: N/A
391
392
393
394 Hosts
395 -----
396
397 Ceph expects at least one monitor declared in the Ceph configuration file, with
398 a ``mon addr`` setting under each declared monitor. Ceph expects a ``host``
399 setting under each declared monitor, metadata server and OSD in the Ceph
400 configuration file. Optionally, a monitor can be assigned with a priority, and
401 the clients will always connect to the monitor with lower value of priority if
402 specified.
403
404
405 ``mon addr``
406
407 :Description: A list of ``{hostname}:{port}`` entries that clients can use to
408 connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]``
409 sections.
410
411 :Type: String
412 :Required: No
413 :Default: N/A
414
415 ``mon priority``
416
417 :Description: The priority of the declared monitor, the lower value the more
418 prefered when a client selects a monitor when trying to connect
419 to the cluster.
420
421 :Type: Unsigned 16-bit Integer
422 :Required: No
423 :Default: 0
424
425 ``host``
426
427 :Description: The hostname. Use this setting for specific daemon instances
428 (e.g., ``[osd.0]``).
429
430 :Type: String
431 :Required: Yes, for daemon instances.
432 :Default: ``localhost``
433
434 .. tip:: Do not use ``localhost``. To get your host name, execute
435 ``hostname -s`` on your command line and use the name of your host
436 (to the first period, not the fully-qualified domain name).
437
438 .. important:: You should not specify any value for ``host`` when using a third
439 party deployment system that retrieves the host name for you.
440
441
442
443 TCP
444 ---
445
446 Ceph disables TCP buffering by default.
447
448
449 ``ms tcp nodelay``
450
451 :Description: Ceph enables ``ms tcp nodelay`` so that each request is sent
452 immediately (no buffering). Disabling `Nagle's algorithm`_
453 increases network traffic, which can introduce latency. If you
454 experience large numbers of small packets, you may try
455 disabling ``ms tcp nodelay``.
456
457 :Type: Boolean
458 :Required: No
459 :Default: ``true``
460
461
462
463 ``ms tcp rcvbuf``
464
465 :Description: The size of the socket buffer on the receiving end of a network
466 connection. Disable by default.
467
468 :Type: 32-bit Integer
469 :Required: No
470 :Default: ``0``
471
472
473
474 ``ms tcp read timeout``
475
476 :Description: If a client or daemon makes a request to another Ceph daemon and
477 does not drop an unused connection, the ``ms tcp read timeout``
478 defines the connection as idle after the specified number
479 of seconds.
480
481 :Type: Unsigned 64-bit Integer
482 :Required: No
483 :Default: ``900`` 15 minutes.
484
485
486
487 .. _Scalability and High Availability: ../../../architecture#scalability-and-high-availability
488 .. _Hardware Recommendations - Networks: ../../../start/hardware-recommendations#networks
489 .. _Ceph configuration file: ../../../start/quick-ceph-deploy/#create-a-cluster
490 .. _hardware recommendations: ../../../start/hardware-recommendations
491 .. _Monitor / OSD Interaction: ../mon-osd-interaction
492 .. _Message Signatures: ../auth-config-ref#signatures
493 .. _CIDR: http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing
494 .. _Nagle's Algorithm: http://en.wikipedia.org/wiki/Nagle's_algorithm