]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[sysadmin_network_configuration]] |
0bcd1f7f DM |
2 | Network Configuration |
3 | --------------------- | |
5f09af76 DM |
4 | ifdef::wiki[] |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
7 | ||
a22d7c24 | 8 | Network configuration can be done either via the GUI, or by manually |
05213009 EK |
9 | editing the file `/etc/network/interfaces`, which contains the |
10 | whole network configuration. The `interfaces(5)` manual page contains the | |
11 | complete format description. All {pve} tools try hard to keep direct | |
12 | user modifications, but using the GUI is still preferable, because it | |
13 | protects you from errors. | |
0bcd1f7f | 14 | |
a22d7c24 | 15 | Once the network is configured, you can use the Debian traditional tools `ifup` |
05213009 | 16 | and `ifdown` commands to bring interfaces up and down. |
0bcd1f7f DM |
17 | |
18 | NOTE: {pve} does not write changes directly to | |
8c1189b6 FG |
19 | `/etc/network/interfaces`. Instead, we write into a temporary file |
20 | called `/etc/network/interfaces.new`, and commit those changes when | |
0bcd1f7f DM |
21 | you reboot the node. |
22 | ||
0bcd1f7f DM |
23 | Naming Conventions |
24 | ~~~~~~~~~~~~~~~~~~ | |
25 | ||
26 | We currently use the following naming conventions for device names: | |
27 | ||
05213009 EK |
28 | * Ethernet devices: en*, systemd network interface names. This naming scheme is |
29 | used for new {pve} installations since version 5.0. | |
7a0d4784 | 30 | |
05213009 EK |
31 | * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) This naming |
32 | scheme is used for {pve} hosts which were installed before the 5.0 | |
33 | release. When upgrading to 5.0, the names are kept as-is. | |
0bcd1f7f DM |
34 | |
35 | * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`) | |
36 | ||
37 | * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...) | |
38 | ||
39 | * VLANs: Simply add the VLAN number to the device name, | |
7a0d4784 | 40 | separated by a period (`eno1.50`, `bond1.30`) |
0bcd1f7f DM |
41 | |
42 | This makes it easier to debug networks problems, because the device | |
05213009 | 43 | name implies the device type. |
cc3cb912 | 44 | |
7a0d4784 WL |
45 | Systemd Network Interface Names |
46 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
47 | ||
8116dea5 DM |
48 | Systemd uses the two character prefix 'en' for Ethernet network |
49 | devices. The next characters depends on the device driver and the fact | |
cc3cb912 | 50 | which schema matches first. |
7a0d4784 WL |
51 | |
52 | * o<index>[n<phys_port_name>|d<dev_port>] — devices on board | |
53 | ||
54 | * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id | |
55 | ||
56 | * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id | |
57 | ||
58 | * x<MAC> — device by MAC address | |
59 | ||
cc3cb912 | 60 | The most common patterns are: |
7a0d4784 WL |
61 | |
62 | * eno1 — is the first on board NIC | |
63 | ||
64 | * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1. | |
65 | ||
cc3cb912 DM |
66 | For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names]. |
67 | ||
05213009 EK |
68 | Choosing a network configuration |
69 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
70 | ||
a22d7c24 | 71 | Depending on your current network organization and your resources you can |
05213009 EK |
72 | choose either a bridged, routed, or masquerading networking setup. |
73 | ||
74 | {pve} server in a private LAN, using an external gateway to reach the internet | |
75 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
76 | ||
a22d7c24 | 77 | The *Bridged* model makes the most sense in this case, and this is also |
05213009 | 78 | the default mode on new {pve} installations. |
a22d7c24 SR |
79 | Each of your Guest system will have a virtual interface attached to the |
80 | {pve} bridge. This is similar in effect to having the Guest network card | |
05213009 EK |
81 | directly connected to a new switch on your LAN, the {pve} host playing the role |
82 | of the switch. | |
83 | ||
84 | {pve} server at hosting provider, with public IP ranges for Guests | |
85 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
86 | ||
87 | For this setup, you can use either a *Bridged* or *Routed* model, depending on | |
88 | what your provider allows. | |
89 | ||
90 | {pve} server at hosting provider, with a single public IP address | |
91 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
92 | ||
93 | In that case the only way to get outgoing network accesses for your guest | |
a22d7c24 | 94 | systems is to use *Masquerading*. For incoming network access to your guests, |
05213009 EK |
95 | you will need to configure *Port Forwarding*. |
96 | ||
97 | For further flexibility, you can configure | |
98 | VLANs (IEEE 802.1q) and network bonding, also known as "link | |
99 | aggregation". That way it is possible to build complex and flexible | |
100 | virtual networks. | |
7a0d4784 | 101 | |
0bcd1f7f DM |
102 | Default Configuration using a Bridge |
103 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
104 | ||
50f88938 | 105 | [thumbnail="default-network-setup-bridge.svg"] |
05213009 | 106 | Bridges are like physical network switches implemented in software. |
a22d7c24 | 107 | All VMs can share a single bridge, or you can create multiple bridges to |
05213009 EK |
108 | separate network domains. Each host can have up to 4094 bridges. |
109 | ||
0bcd1f7f | 110 | The installation program creates a single bridge named `vmbr0`, which |
04e8476d EK |
111 | is connected to the first Ethernet card. The corresponding |
112 | configuration in `/etc/network/interfaces` might look like this: | |
0bcd1f7f DM |
113 | |
114 | ---- | |
115 | auto lo | |
116 | iface lo inet loopback | |
117 | ||
7a0d4784 | 118 | iface eno1 inet manual |
0bcd1f7f DM |
119 | |
120 | auto vmbr0 | |
121 | iface vmbr0 inet static | |
122 | address 192.168.10.2 | |
123 | netmask 255.255.255.0 | |
124 | gateway 192.168.10.1 | |
7a0d4784 | 125 | bridge_ports eno1 |
0bcd1f7f DM |
126 | bridge_stp off |
127 | bridge_fd 0 | |
128 | ---- | |
129 | ||
130 | Virtual machines behave as if they were directly connected to the | |
131 | physical network. The network, in turn, sees each virtual machine as | |
132 | having its own MAC, even though there is only one network cable | |
133 | connecting all of these VMs to the network. | |
134 | ||
0bcd1f7f DM |
135 | Routed Configuration |
136 | ~~~~~~~~~~~~~~~~~~~~ | |
137 | ||
138 | Most hosting providers do not support the above setup. For security | |
139 | reasons, they disable networking as soon as they detect multiple MAC | |
140 | addresses on a single interface. | |
141 | ||
552e175d | 142 | TIP: Some providers allows you to register additional MACs on their |
0bcd1f7f DM |
143 | management interface. This avoids the problem, but is clumsy to |
144 | configure because you need to register a MAC for each of your VMs. | |
145 | ||
8c1189b6 | 146 | You can avoid the problem by ``routing'' all traffic via a single |
0bcd1f7f DM |
147 | interface. This makes sure that all network packets use the same MAC |
148 | address. | |
149 | ||
50f88938 | 150 | [thumbnail="default-network-setup-routed.svg"] |
05213009 | 151 | A common scenario is that you have a public IP (assume `198.51.100.5` |
0bcd1f7f | 152 | for this example), and an additional IP block for your VMs |
05213009 | 153 | (`203.0.113.16/29`). We recommend the following setup for such |
0bcd1f7f DM |
154 | situations: |
155 | ||
156 | ---- | |
157 | auto lo | |
158 | iface lo inet loopback | |
159 | ||
7a0d4784 WL |
160 | auto eno1 |
161 | iface eno1 inet static | |
05213009 | 162 | address 198.51.100.5 |
0bcd1f7f | 163 | netmask 255.255.255.0 |
05213009 | 164 | gateway 198.51.100.1 |
1ed90852 | 165 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward |
7a0d4784 | 166 | post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp |
0bcd1f7f DM |
167 | |
168 | ||
169 | auto vmbr0 | |
170 | iface vmbr0 inet static | |
05213009 EK |
171 | address 203.0.113.17 |
172 | netmask 255.255.255.248 | |
0bcd1f7f DM |
173 | bridge_ports none |
174 | bridge_stp off | |
175 | bridge_fd 0 | |
176 | ---- | |
177 | ||
178 | ||
8c1189b6 FG |
179 | Masquerading (NAT) with `iptables` |
180 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
0bcd1f7f | 181 | |
05213009 EK |
182 | Masquerading allows guests having only a private IP address to access the |
183 | network by using the host IP address for outgoing traffic. Each outgoing | |
184 | packet is rewritten by `iptables` to appear as originating from the host, | |
185 | and responses are rewritten accordingly to be routed to the original sender. | |
0bcd1f7f DM |
186 | |
187 | ---- | |
188 | auto lo | |
189 | iface lo inet loopback | |
190 | ||
05213009 | 191 | auto eno1 |
470d4313 | 192 | #real IP address |
7a0d4784 | 193 | iface eno1 inet static |
05213009 | 194 | address 198.51.100.5 |
0bcd1f7f | 195 | netmask 255.255.255.0 |
05213009 | 196 | gateway 198.51.100.1 |
0bcd1f7f DM |
197 | |
198 | auto vmbr0 | |
199 | #private sub network | |
200 | iface vmbr0 inet static | |
201 | address 10.10.10.1 | |
202 | netmask 255.255.255.0 | |
203 | bridge_ports none | |
204 | bridge_stp off | |
205 | bridge_fd 0 | |
206 | ||
207 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward | |
7a0d4784 WL |
208 | post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE |
209 | post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE | |
0bcd1f7f DM |
210 | ---- |
211 | ||
b4c06a93 WL |
212 | |
213 | Linux Bond | |
214 | ~~~~~~~~~~ | |
215 | ||
3eafe338 WL |
216 | Bonding (also called NIC teaming or Link Aggregation) is a technique |
217 | for binding multiple NIC's to a single network device. It is possible | |
218 | to achieve different goals, like make the network fault-tolerant, | |
219 | increase the performance or both together. | |
220 | ||
221 | High-speed hardware like Fibre Channel and the associated switching | |
222 | hardware can be quite expensive. By doing link aggregation, two NICs | |
223 | can appear as one logical interface, resulting in double speed. This | |
224 | is a native Linux kernel feature that is supported by most | |
225 | switches. If your nodes have multiple Ethernet ports, you can | |
226 | distribute your points of failure by running network cables to | |
227 | different switches and the bonded connection will failover to one | |
228 | cable or the other in case of network trouble. | |
229 | ||
230 | Aggregated links can improve live-migration delays and improve the | |
231 | speed of replication of data between Proxmox VE Cluster nodes. | |
b4c06a93 WL |
232 | |
233 | There are 7 modes for bonding: | |
234 | ||
235 | * *Round-robin (balance-rr):* Transmit network packets in sequential | |
236 | order from the first available network interface (NIC) slave through | |
237 | the last. This mode provides load balancing and fault tolerance. | |
238 | ||
239 | * *Active-backup (active-backup):* Only one NIC slave in the bond is | |
240 | active. A different slave becomes active if, and only if, the active | |
241 | slave fails. The single logical bonded interface's MAC address is | |
242 | externally visible on only one NIC (port) to avoid distortion in the | |
243 | network switch. This mode provides fault tolerance. | |
244 | ||
245 | * *XOR (balance-xor):* Transmit network packets based on [(source MAC | |
246 | address XOR'd with destination MAC address) modulo NIC slave | |
247 | count]. This selects the same NIC slave for each destination MAC | |
248 | address. This mode provides load balancing and fault tolerance. | |
249 | ||
250 | * *Broadcast (broadcast):* Transmit network packets on all slave | |
251 | network interfaces. This mode provides fault tolerance. | |
252 | ||
253 | * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates | |
254 | aggregation groups that share the same speed and duplex | |
255 | settings. Utilizes all slave network interfaces in the active | |
256 | aggregator group according to the 802.3ad specification. | |
257 | ||
258 | * *Adaptive transmit load balancing (balance-tlb):* Linux bonding | |
259 | driver mode that does not require any special network-switch | |
260 | support. The outgoing network packet traffic is distributed according | |
261 | to the current load (computed relative to the speed) on each network | |
262 | interface slave. Incoming traffic is received by one currently | |
263 | designated slave network interface. If this receiving slave fails, | |
264 | another slave takes over the MAC address of the failed receiving | |
265 | slave. | |
266 | ||
e60ce90c | 267 | * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive |
b4c06a93 WL |
268 | load balancing (rlb) for IPV4 traffic, and does not require any |
269 | special network switch support. The receive load balancing is achieved | |
270 | by ARP negotiation. The bonding driver intercepts the ARP Replies sent | |
271 | by the local system on their way out and overwrites the source | |
272 | hardware address with the unique hardware address of one of the NIC | |
273 | slaves in the single logical bonded interface such that different | |
274 | network-peers use different MAC addresses for their network packet | |
275 | traffic. | |
276 | ||
649098a6 | 277 | If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using |
a22d7c24 | 278 | the corresponding bonding mode (802.3ad). Otherwise you should generally use the |
649098a6 EK |
279 | active-backup mode. + |
280 | // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html | |
281 | If you intend to run your cluster network on the bonding interfaces, then you | |
282 | have to use active-passive mode on the bonding interfaces, other modes are | |
283 | unsupported. | |
b4c06a93 | 284 | |
cd1de2c2 WL |
285 | The following bond configuration can be used as distributed/shared |
286 | storage network. The benefit would be that you get more speed and the | |
287 | network will be fault-tolerant. | |
288 | ||
b4c06a93 WL |
289 | .Example: Use bond with fixed IP address |
290 | ---- | |
291 | auto lo | |
292 | iface lo inet loopback | |
293 | ||
7a0d4784 | 294 | iface eno1 inet manual |
b4c06a93 | 295 | |
7a0d4784 | 296 | iface eno2 inet manual |
b4c06a93 WL |
297 | |
298 | auto bond0 | |
299 | iface bond0 inet static | |
7a0d4784 | 300 | slaves eno1 eno2 |
b4c06a93 WL |
301 | address 192.168.1.2 |
302 | netmask 255.255.255.0 | |
303 | bond_miimon 100 | |
304 | bond_mode 802.3ad | |
305 | bond_xmit_hash_policy layer2+3 | |
306 | ||
307 | auto vmbr0 | |
308 | iface vmbr0 inet static | |
309 | address 10.10.10.2 | |
310 | netmask 255.255.255.0 | |
7ea42266 | 311 | gateway 10.10.10.1 |
7a0d4784 | 312 | bridge_ports eno1 |
b4c06a93 WL |
313 | bridge_stp off |
314 | bridge_fd 0 | |
315 | ||
316 | ---- | |
317 | ||
cd1de2c2 | 318 | |
50f88938 | 319 | [thumbnail="default-network-setup-bond.svg"] |
cd1de2c2 WL |
320 | Another possibility it to use the bond directly as bridge port. |
321 | This can be used to make the guest network fault-tolerant. | |
322 | ||
323 | .Example: Use a bond as bridge port | |
b4c06a93 WL |
324 | ---- |
325 | auto lo | |
326 | iface lo inet loopback | |
327 | ||
7a0d4784 | 328 | iface eno1 inet manual |
b4c06a93 | 329 | |
7a0d4784 | 330 | iface eno2 inet manual |
b4c06a93 WL |
331 | |
332 | auto bond0 | |
470d4313 | 333 | iface bond0 inet manual |
7a0d4784 | 334 | slaves eno1 eno2 |
b4c06a93 WL |
335 | bond_miimon 100 |
336 | bond_mode 802.3ad | |
337 | bond_xmit_hash_policy layer2+3 | |
338 | ||
339 | auto vmbr0 | |
340 | iface vmbr0 inet static | |
341 | address 10.10.10.2 | |
342 | netmask 255.255.255.0 | |
7ea42266 | 343 | gateway 10.10.10.1 |
b4c06a93 WL |
344 | bridge_ports bond0 |
345 | bridge_stp off | |
346 | bridge_fd 0 | |
347 | ||
348 | ---- | |
349 | ||
61105e42 | 350 | |
94fd8ea5 WL |
351 | VLAN 802.1Q |
352 | ~~~~~~~~~~~ | |
353 | ||
4d8af129 DM |
354 | A virtual LAN (VLAN) is a broadcast domain that is partitioned and |
355 | isolated in the network at layer two. So it is possible to have | |
356 | multiple networks (4096) in a physical network, each independent of | |
357 | the other ones. | |
358 | ||
61105e42 | 359 | Each VLAN network is identified by a number often called 'tag'. |
4d8af129 DM |
360 | Network packages are then 'tagged' to identify which virtual network |
361 | they belong to. | |
94fd8ea5 | 362 | |
94fd8ea5 | 363 | |
4d8af129 DM |
364 | VLAN for Guest Networks |
365 | ^^^^^^^^^^^^^^^^^^^^^^^ | |
94fd8ea5 | 366 | |
4d8af129 DM |
367 | {pve} supports this setup out of the box. You can specify the VLAN tag |
368 | when you create a VM. The VLAN tag is part of the guest network | |
a22d7c24 | 369 | configuration. The networking layer supports different modes to |
4d8af129 | 370 | implement VLANs, depending on the bridge configuration: |
94fd8ea5 | 371 | |
4d8af129 | 372 | * *VLAN awareness on the Linux bridge:* |
94fd8ea5 | 373 | In this case, each guest's virtual network card is assigned to a VLAN tag, |
4d8af129 | 374 | which is transparently supported by the Linux bridge. |
a22d7c24 | 375 | Trunk mode is also possible, but that makes configuration |
94fd8ea5 WL |
376 | in the guest necessary. |
377 | ||
378 | * *"traditional" VLAN on the Linux bridge:* | |
379 | In contrast to the VLAN awareness method, this method is not transparent | |
380 | and creates a VLAN device with associated bridge for each VLAN. | |
a22d7c24 SR |
381 | That is, creating a guest on VLAN 5 for example, would create two |
382 | interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs. | |
94fd8ea5 | 383 | |
4d8af129 DM |
384 | * *Open vSwitch VLAN:* |
385 | This mode uses the OVS VLAN feature. | |
386 | ||
a22d7c24 | 387 | * *Guest configured VLAN:* |
4d8af129 DM |
388 | VLANs are assigned inside the guest. In this case, the setup is |
389 | completely done inside the guest and can not be influenced from the | |
390 | outside. The benefit is that you can use more than one VLAN on a | |
391 | single virtual NIC. | |
392 | ||
393 | ||
394 | VLAN on the Host | |
395 | ^^^^^^^^^^^^^^^^ | |
94fd8ea5 | 396 | |
4d8af129 DM |
397 | To allow host communication with an isolated network. It is possible |
398 | to apply VLAN tags to any network device (NIC, Bond, Bridge). In | |
399 | general, you should configure the VLAN on the interface with the least | |
94fd8ea5 WL |
400 | abstraction layers between itself and the physical NIC. |
401 | ||
402 | For example, in a default configuration where you want to place | |
403 | the host management address on a separate VLAN. | |
404 | ||
94fd8ea5 | 405 | |
038dc7df | 406 | .Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge |
94fd8ea5 WL |
407 | ---- |
408 | auto lo | |
409 | iface lo inet loopback | |
410 | ||
411 | iface eno1 inet manual | |
412 | ||
413 | iface eno1.5 inet manual | |
414 | ||
415 | auto vmbr0v5 | |
416 | iface vmbr0v5 inet static | |
417 | address 10.10.10.2 | |
418 | netmask 255.255.255.0 | |
419 | gateway 10.10.10.1 | |
420 | bridge_ports eno1.5 | |
421 | bridge_stp off | |
422 | bridge_fd 0 | |
423 | ||
424 | auto vmbr0 | |
425 | iface vmbr0 inet manual | |
426 | bridge_ports eno1 | |
427 | bridge_stp off | |
428 | bridge_fd 0 | |
429 | ||
430 | ---- | |
431 | ||
038dc7df AD |
432 | .Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge |
433 | ---- | |
434 | auto lo | |
435 | iface lo inet loopback | |
436 | ||
437 | iface eno1 inet manual | |
438 | ||
439 | ||
440 | auto vmbr0.5 | |
441 | iface vmbr0.5 inet static | |
442 | address 10.10.10.2 | |
443 | netmask 255.255.255.0 | |
444 | gateway 10.10.10.1 | |
445 | ||
446 | auto vmbr0 | |
447 | iface vmbr0 inet manual | |
448 | bridge_ports eno1 | |
449 | bridge_stp off | |
450 | bridge_fd 0 | |
451 | bridge_vlan_aware yes | |
452 | ---- | |
453 | ||
94fd8ea5 WL |
454 | The next example is the same setup but a bond is used to |
455 | make this network fail-safe. | |
456 | ||
038dc7df | 457 | .Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge |
94fd8ea5 WL |
458 | ---- |
459 | auto lo | |
460 | iface lo inet loopback | |
461 | ||
462 | iface eno1 inet manual | |
463 | ||
464 | iface eno2 inet manual | |
465 | ||
466 | auto bond0 | |
467 | iface bond0 inet manual | |
468 | slaves eno1 eno2 | |
469 | bond_miimon 100 | |
470 | bond_mode 802.3ad | |
471 | bond_xmit_hash_policy layer2+3 | |
472 | ||
473 | iface bond0.5 inet manual | |
474 | ||
475 | auto vmbr0v5 | |
476 | iface vmbr0v5 inet static | |
477 | address 10.10.10.2 | |
478 | netmask 255.255.255.0 | |
479 | gateway 10.10.10.1 | |
480 | bridge_ports bond0.5 | |
481 | bridge_stp off | |
482 | bridge_fd 0 | |
483 | ||
484 | auto vmbr0 | |
485 | iface vmbr0 inet manual | |
486 | bridge_ports bond0 | |
487 | bridge_stp off | |
488 | bridge_fd 0 | |
489 | ||
490 | ---- | |
491 | ||
0bcd1f7f DM |
492 | //// |
493 | TODO: explain IPv6 support? | |
470d4313 | 494 | TODO: explain OVS |
0bcd1f7f | 495 | //// |