]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | ================================= |
2 | Network Configuration Reference | |
3 | ================================= | |
4 | ||
5 | Network configuration is critical for building a high performance :term:`Ceph | |
6 | Storage Cluster`. The Ceph Storage Cluster does not perform request routing or | |
7 | dispatching on behalf of the :term:`Ceph Client`. Instead, Ceph Clients make | |
8 | requests directly to Ceph OSD Daemons. Ceph OSD Daemons perform data replication | |
9 | on behalf of Ceph Clients, which means replication and other factors impose | |
10 | additional loads on Ceph Storage Cluster networks. | |
11 | ||
12 | Our Quick Start configurations provide a trivial `Ceph configuration file`_ that | |
13 | sets monitor IP addresses and daemon host names only. Unless you specify a | |
14 | cluster network, Ceph assumes a single "public" network. Ceph functions just | |
15 | fine with a public network only, but you may see significant performance | |
16 | improvement with a second "cluster" network in a large cluster. | |
17 | ||
11fdf7f2 TL |
18 | It is possible to run a Ceph Storage Cluster with two networks: a public |
19 | (front-side) network and a cluster (back-side) network. However, this approach | |
20 | complicates network configuration (both hardware and software) and does not usually have a significant impact on overall performance. For this reason, we generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded. | |
21 | ||
22 | If, despite the complexity, one still wishes to use two networks, each | |
23 | :term:`Ceph Node` will need to have more than one NIC. See `Hardware | |
24 | Recommendations - Networks`_ for additional details. | |
7c673cae FG |
25 | |
26 | .. ditaa:: | |
27 | +-------------+ | |
28 | | Ceph Client | | |
29 | +----*--*-----+ | |
30 | | ^ | |
31 | Request | : Response | |
32 | v | | |
33 | /----------------------------------*--*-------------------------------------\ | |
34 | | Public Network | | |
35 | \---*--*------------*--*-------------*--*------------*--*------------*--*---/ | |
36 | ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | |
37 | | | | | | | | | | | | |
38 | | : | : | : | : | : | |
39 | v v v v v v v v v v | |
40 | +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+ | |
41 | | Ceph MON | | Ceph MDS | | Ceph OSD | | Ceph OSD | | Ceph OSD | | |
42 | +----------+ +----------+ +---*--*---+ +---*--*---+ +---*--*---+ | |
43 | ^ ^ ^ ^ ^ ^ | |
44 | The cluster network relieves | | | | | | | |
45 | OSD replication and heartbeat | : | : | : | |
46 | traffic from the public network. v v v v v v | |
47 | /------------------------------------*--*------------*--*------------*--*---\ | |
48 | | cCCC Cluster Network | | |
49 | \---------------------------------------------------------------------------/ | |
50 | ||
51 | ||
7c673cae FG |
52 | IP Tables |
53 | ========= | |
54 | ||
55 | By default, daemons `bind`_ to ports within the ``6800:7300`` range. You may | |
56 | configure this range at your discretion. Before configuring your IP tables, | |
57 | check the default ``iptables`` configuration. | |
58 | ||
59 | sudo iptables -L | |
60 | ||
61 | Some Linux distributions include rules that reject all inbound requests | |
62 | except SSH from all network interfaces. For example:: | |
63 | ||
64 | REJECT all -- anywhere anywhere reject-with icmp-host-prohibited | |
65 | ||
66 | You will need to delete these rules on both your public and cluster networks | |
67 | initially, and replace them with appropriate rules when you are ready to | |
68 | harden the ports on your Ceph Nodes. | |
69 | ||
70 | ||
71 | Monitor IP Tables | |
72 | ----------------- | |
73 | ||
11fdf7f2 TL |
74 | Ceph Monitors listen on ports ``3300`` and ``6789`` by |
75 | default. Additionally, Ceph Monitors always operate on the public | |
76 | network. When you add the rule using the example below, make sure you | |
77 | replace ``{iface}`` with the public network interface (e.g., ``eth0``, | |
78 | ``eth1``, etc.), ``{ip-address}`` with the IP address of the public | |
79 | network and ``{netmask}`` with the netmask for the public network. :: | |
7c673cae FG |
80 | |
81 | sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT | |
82 | ||
83 | ||
11fdf7f2 TL |
84 | MDS and Manager IP Tables |
85 | ------------------------- | |
7c673cae | 86 | |
11fdf7f2 TL |
87 | A :term:`Ceph Metadata Server` or :term:`Ceph Manager` listens on the first |
88 | available port on the public network beginning at port 6800. Note that this | |
89 | behavior is not deterministic, so if you are running more than one OSD or MDS | |
90 | on the same host, or if you restart the daemons within a short window of time, | |
91 | the daemons will bind to higher ports. You should open the entire 6800-7300 | |
92 | range by default. When you add the rule using the example below, make sure | |
93 | you replace ``{iface}`` with the public network interface (e.g., ``eth0``, | |
94 | ``eth1``, etc.), ``{ip-address}`` with the IP address of the public network | |
95 | and ``{netmask}`` with the netmask of the public network. | |
7c673cae FG |
96 | |
97 | For example:: | |
98 | ||
99 | sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT | |
100 | ||
101 | ||
102 | OSD IP Tables | |
103 | ------------- | |
104 | ||
105 | By default, Ceph OSD Daemons `bind`_ to the first available ports on a Ceph Node | |
106 | beginning at port 6800. Note that this behavior is not deterministic, so if you | |
107 | are running more than one OSD or MDS on the same host, or if you restart the | |
108 | daemons within a short window of time, the daemons will bind to higher ports. | |
109 | Each Ceph OSD Daemon on a Ceph Node may use up to four ports: | |
110 | ||
111 | #. One for talking to clients and monitors. | |
112 | #. One for sending data to other OSDs. | |
113 | #. Two for heartbeating on each interface. | |
114 | ||
115 | .. ditaa:: | |
116 | /---------------\ | |
117 | | OSD | | |
118 | | +---+----------------+-----------+ | |
119 | | | Clients & Monitors | Heartbeat | | |
120 | | +---+----------------+-----------+ | |
121 | | | | |
122 | | +---+----------------+-----------+ | |
123 | | | Data Replication | Heartbeat | | |
124 | | +---+----------------+-----------+ | |
125 | | cCCC | | |
126 | \---------------/ | |
127 | ||
128 | When a daemon fails and restarts without letting go of the port, the restarted | |
129 | daemon will bind to a new port. You should open the entire 6800-7300 port range | |
130 | to handle this possibility. | |
131 | ||
132 | If you set up separate public and cluster networks, you must add rules for both | |
133 | the public network and the cluster network, because clients will connect using | |
134 | the public network and other Ceph OSD Daemons will connect using the cluster | |
135 | network. When you add the rule using the example below, make sure you replace | |
136 | ``{iface}`` with the network interface (e.g., ``eth0``, ``eth1``, etc.), | |
137 | ``{ip-address}`` with the IP address and ``{netmask}`` with the netmask of the | |
138 | public or cluster network. For example:: | |
139 | ||
140 | sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT | |
141 | ||
142 | .. tip:: If you run Ceph Metadata Servers on the same Ceph Node as the | |
143 | Ceph OSD Daemons, you can consolidate the public network configuration step. | |
144 | ||
145 | ||
146 | Ceph Networks | |
147 | ============= | |
148 | ||
149 | To configure Ceph networks, you must add a network configuration to the | |
150 | ``[global]`` section of the configuration file. Our 5-minute Quick Start | |
151 | provides a trivial `Ceph configuration file`_ that assumes one public network | |
152 | with client and server on the same network and subnet. Ceph functions just fine | |
153 | with a public network only. However, Ceph allows you to establish much more | |
154 | specific criteria, including multiple IP network and subnet masks for your | |
155 | public network. You can also establish a separate cluster network to handle OSD | |
156 | heartbeat, object replication and recovery traffic. Don't confuse the IP | |
157 | addresses you set in your configuration with the public-facing IP addresses | |
158 | network clients may use to access your service. Typical internal IP networks are | |
159 | often ``192.168.0.0`` or ``10.0.0.0``. | |
160 | ||
161 | .. tip:: If you specify more than one IP address and subnet mask for | |
162 | either the public or the cluster network, the subnets within the network | |
163 | must be capable of routing to each other. Additionally, make sure you | |
164 | include each IP address/subnet in your IP tables and open ports for them | |
165 | as necessary. | |
166 | ||
167 | .. note:: Ceph uses `CIDR`_ notation for subnets (e.g., ``10.0.0.0/24``). | |
168 | ||
c07f9fc5 | 169 | When you have configured your networks, you may restart your cluster or restart |
7c673cae FG |
170 | each daemon. Ceph daemons bind dynamically, so you do not have to restart the |
171 | entire cluster at once if you change your network configuration. | |
172 | ||
173 | ||
174 | Public Network | |
175 | -------------- | |
176 | ||
177 | To configure a public network, add the following option to the ``[global]`` | |
178 | section of your Ceph configuration file. | |
179 | ||
180 | .. code-block:: ini | |
181 | ||
182 | [global] | |
11fdf7f2 | 183 | # ... elided configuration |
7c673cae FG |
184 | public network = {public-network/netmask} |
185 | ||
186 | ||
187 | Cluster Network | |
188 | --------------- | |
189 | ||
190 | If you declare a cluster network, OSDs will route heartbeat, object replication | |
191 | and recovery traffic over the cluster network. This may improve performance | |
192 | compared to using a single network. To configure a cluster network, add the | |
193 | following option to the ``[global]`` section of your Ceph configuration file. | |
194 | ||
195 | .. code-block:: ini | |
196 | ||
197 | [global] | |
11fdf7f2 | 198 | # ... elided configuration |
7c673cae FG |
199 | cluster network = {cluster-network/netmask} |
200 | ||
201 | We prefer that the cluster network is **NOT** reachable from the public network | |
202 | or the Internet for added security. | |
203 | ||
204 | ||
205 | Ceph Daemons | |
206 | ============ | |
207 | ||
11fdf7f2 | 208 | The monitor daemons are each configured to bind to a specific IP address. These addresses are normally configured by your deployment tool. Other components in the Ceph system discover the monitors via the ``mon host`` configuration option, normally specified in the ``[global]`` section of the ``ceph.conf`` file. |
7c673cae FG |
209 | |
210 | .. code-block:: ini | |
211 | ||
11fdf7f2 TL |
212 | [global] |
213 | mon host = 10.0.0.2, 10.0.0.3, 10.0.0.4 | |
7c673cae | 214 | |
11fdf7f2 TL |
215 | The ``mon host`` value can be a list of IP addresses or a name that is |
216 | looked up via DNS. In the case of a DNS name with multiple A or AAAA | |
217 | records, all records are probed in order to discover a monitor. Once | |
218 | one monitor is reached, all other current monitors are discovered, so | |
219 | the ``mon host`` configuration option only needs to be sufficiently up | |
220 | to date such that a client can reach one monitor that is currently online. | |
7c673cae | 221 | |
11fdf7f2 TL |
222 | The MGR, OSD, and MDS daemons will bind to any available address and |
223 | do not require any special configuration. However, it is possible to | |
224 | specify a specific IP address for them to bind to with the ``public | |
225 | addr`` (and/or, in the case of OSD daemons, the ``cluster addr``) | |
226 | configuration option. For example, | |
7c673cae FG |
227 | |
228 | .. code-block:: ini | |
229 | ||
230 | [osd.0] | |
231 | public addr = {host-public-ip-address} | |
232 | cluster addr = {host-cluster-ip-address} | |
233 | ||
7c673cae FG |
234 | .. topic:: One NIC OSD in a Two Network Cluster |
235 | ||
236 | Generally, we do not recommend deploying an OSD host with a single NIC in a | |
237 | cluster with two networks. However, you may accomplish this by forcing the | |
238 | OSD host to operate on the public network by adding a ``public addr`` entry | |
239 | to the ``[osd.n]`` section of the Ceph configuration file, where ``n`` | |
240 | refers to the number of the OSD with one NIC. Additionally, the public | |
241 | network and cluster network must be able to route traffic to each other, | |
242 | which we don't recommend for security reasons. | |
243 | ||
244 | ||
245 | Network Config Settings | |
246 | ======================= | |
247 | ||
248 | Network configuration settings are not required. Ceph assumes a public network | |
249 | with all hosts operating on it unless you specifically configure a cluster | |
250 | network. | |
251 | ||
252 | ||
253 | Public Network | |
254 | -------------- | |
255 | ||
256 | The public network configuration allows you specifically define IP addresses | |
257 | and subnets for the public network. You may specifically assign static IP | |
258 | addresses or override ``public network`` settings using the ``public addr`` | |
259 | setting for a specific daemon. | |
260 | ||
261 | ``public network`` | |
262 | ||
263 | :Description: The IP address and netmask of the public (front-side) network | |
264 | (e.g., ``192.168.0.0/24``). Set in ``[global]``. You may specify | |
9f95a23c | 265 | comma-separated subnets. |
7c673cae FG |
266 | |
267 | :Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]`` | |
268 | :Required: No | |
269 | :Default: N/A | |
270 | ||
271 | ||
272 | ``public addr`` | |
273 | ||
274 | :Description: The IP address for the public (front-side) network. | |
275 | Set for each daemon. | |
276 | ||
277 | :Type: IP Address | |
278 | :Required: No | |
279 | :Default: N/A | |
280 | ||
281 | ||
282 | ||
283 | Cluster Network | |
284 | --------------- | |
285 | ||
286 | The cluster network configuration allows you to declare a cluster network, and | |
287 | specifically define IP addresses and subnets for the cluster network. You may | |
288 | specifically assign static IP addresses or override ``cluster network`` | |
289 | settings using the ``cluster addr`` setting for specific OSD daemons. | |
290 | ||
291 | ||
292 | ``cluster network`` | |
293 | ||
294 | :Description: The IP address and netmask of the cluster (back-side) network | |
295 | (e.g., ``10.0.0.0/24``). Set in ``[global]``. You may specify | |
9f95a23c | 296 | comma-separated subnets. |
7c673cae FG |
297 | |
298 | :Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]`` | |
299 | :Required: No | |
300 | :Default: N/A | |
301 | ||
302 | ||
303 | ``cluster addr`` | |
304 | ||
305 | :Description: The IP address for the cluster (back-side) network. | |
306 | Set for each daemon. | |
307 | ||
308 | :Type: Address | |
309 | :Required: No | |
310 | :Default: N/A | |
311 | ||
312 | ||
313 | Bind | |
314 | ---- | |
315 | ||
316 | Bind settings set the default port ranges Ceph OSD and MDS daemons use. The | |
317 | default range is ``6800:7300``. Ensure that your `IP Tables`_ configuration | |
318 | allows you to use the configured port range. | |
319 | ||
320 | You may also enable Ceph daemons to bind to IPv6 addresses instead of IPv4 | |
321 | addresses. | |
322 | ||
323 | ||
324 | ``ms bind port min`` | |
325 | ||
326 | :Description: The minimum port number to which an OSD or MDS daemon will bind. | |
327 | :Type: 32-bit Integer | |
328 | :Default: ``6800`` | |
329 | :Required: No | |
330 | ||
331 | ||
332 | ``ms bind port max`` | |
333 | ||
334 | :Description: The maximum port number to which an OSD or MDS daemon will bind. | |
335 | :Type: 32-bit Integer | |
336 | :Default: ``7300`` | |
337 | :Required: No. | |
338 | ||
339 | ||
340 | ``ms bind ipv6`` | |
341 | ||
342 | :Description: Enables Ceph daemons to bind to IPv6 addresses. Currently the | |
c07f9fc5 | 343 | messenger *either* uses IPv4 or IPv6, but it cannot do both. |
7c673cae FG |
344 | :Type: Boolean |
345 | :Default: ``false`` | |
346 | :Required: No | |
347 | ||
224ce89b WB |
348 | ``public bind addr`` |
349 | ||
350 | :Description: In some dynamic deployments the Ceph MON daemon might bind | |
351 | to an IP address locally that is different from the ``public addr`` | |
352 | advertised to other peers in the network. The environment must ensure | |
11fdf7f2 | 353 | that routing rules are set correctly. If ``public bind addr`` is set |
224ce89b WB |
354 | the Ceph MON daemon will bind to it locally and use ``public addr`` |
355 | in the monmaps to advertise its address to peers. This behavior is limited | |
356 | to the MON daemon. | |
357 | ||
358 | :Type: IP Address | |
359 | :Required: No | |
360 | :Default: N/A | |
361 | ||
7c673cae FG |
362 | |
363 | ||
7c673cae FG |
364 | TCP |
365 | --- | |
366 | ||
367 | Ceph disables TCP buffering by default. | |
368 | ||
369 | ||
370 | ``ms tcp nodelay`` | |
371 | ||
372 | :Description: Ceph enables ``ms tcp nodelay`` so that each request is sent | |
373 | immediately (no buffering). Disabling `Nagle's algorithm`_ | |
374 | increases network traffic, which can introduce latency. If you | |
375 | experience large numbers of small packets, you may try | |
376 | disabling ``ms tcp nodelay``. | |
377 | ||
378 | :Type: Boolean | |
379 | :Required: No | |
380 | :Default: ``true`` | |
381 | ||
382 | ||
383 | ||
384 | ``ms tcp rcvbuf`` | |
385 | ||
386 | :Description: The size of the socket buffer on the receiving end of a network | |
387 | connection. Disable by default. | |
388 | ||
389 | :Type: 32-bit Integer | |
390 | :Required: No | |
391 | :Default: ``0`` | |
392 | ||
393 | ||
394 | ||
395 | ``ms tcp read timeout`` | |
396 | ||
397 | :Description: If a client or daemon makes a request to another Ceph daemon and | |
398 | does not drop an unused connection, the ``ms tcp read timeout`` | |
399 | defines the connection as idle after the specified number | |
400 | of seconds. | |
401 | ||
402 | :Type: Unsigned 64-bit Integer | |
403 | :Required: No | |
404 | :Default: ``900`` 15 minutes. | |
405 | ||
406 | ||
407 | ||
408 | .. _Scalability and High Availability: ../../../architecture#scalability-and-high-availability | |
409 | .. _Hardware Recommendations - Networks: ../../../start/hardware-recommendations#networks | |
410 | .. _Ceph configuration file: ../../../start/quick-ceph-deploy/#create-a-cluster | |
411 | .. _hardware recommendations: ../../../start/hardware-recommendations | |
412 | .. _Monitor / OSD Interaction: ../mon-osd-interaction | |
413 | .. _Message Signatures: ../auth-config-ref#signatures | |
11fdf7f2 TL |
414 | .. _CIDR: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing |
415 | .. _Nagle's Algorithm: https://en.wikipedia.org/wiki/Nagle's_algorithm |