]>
Commit | Line | Data |
---|---|---|
1 | [[sysadmin_network_configuration]] | |
2 | Network Configuration | |
3 | --------------------- | |
4 | ifdef::wiki[] | |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
7 | ||
8 | Network configuration can be done either via the GUI, or by manually | |
9 | editing the file `/etc/network/interfaces`, which contains the | |
10 | whole network configuration. The `interfaces(5)` manual page contains the | |
11 | complete format description. All {pve} tools try hard to keep direct | |
12 | user modifications, but using the GUI is still preferable, because it | |
13 | protects you from errors. | |
14 | ||
15 | Once the network is configured, you can use the Debian traditional tools `ifup` | |
16 | and `ifdown` commands to bring interfaces up and down. | |
17 | ||
18 | Apply Network Changes | |
19 | ~~~~~~~~~~~~~~~~~~~~~ | |
20 | ||
21 | {pve} does not write changes directly to `/etc/network/interfaces`. Instead, we | |
22 | write into a temporary file called `/etc/network/interfaces.new`, this way you | |
23 | can do many related changes at once. This also allows to ensure your changes | |
24 | are correct before applying, as a wrong network configuration may render a node | |
25 | inaccessible. | |
26 | ||
27 | Reboot Node to apply | |
28 | ^^^^^^^^^^^^^^^^^^^^ | |
29 | ||
30 | With the default installed `ifupdown` network managing package you need to | |
31 | reboot to commit any pending network changes. Most of the time, the basic {pve} | |
32 | network setup is stable and does not change often, so rebooting should not be | |
33 | required often. | |
34 | ||
35 | Reload Network with ifupdown2 | |
36 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
37 | ||
38 | With the optional `ifupdown2` network managing package you also can reload the | |
39 | network configuration live, without requiring a reboot. | |
40 | ||
41 | NOTE: 'ifupdown2' cannot understand 'OpenVSwitch' syntax, so reloading is *not* | |
42 | possible if OVS interfaces are configured. | |
43 | ||
44 | Since {pve} 6.1 you can apply pending network changes over the web-interface, | |
45 | using the 'Apply Configuration' button in the 'Network' panel of a node. | |
46 | ||
47 | To install 'ifupdown2' ensure you have the latest {pve} updates installed, then | |
48 | ||
49 | WARNING: installing 'ifupdown2' will remove 'ifupdown', but as the removal | |
50 | scripts of 'ifupdown' before version '0.8.35+pve1' have a issue where network | |
51 | is fully stopped on removal footnote:[Introduced with Debian Buster: | |
52 | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945877] you *must* ensure | |
53 | that you have a up to date 'ifupdown' package version. | |
54 | ||
55 | For the installation itself you can then simply do: | |
56 | ||
57 | apt install ifupdown2 | |
58 | ||
59 | With that you're all set. You can also switch back to the 'ifupdown' variant at | |
60 | any time, if you run into issues. | |
61 | ||
62 | Naming Conventions | |
63 | ~~~~~~~~~~~~~~~~~~ | |
64 | ||
65 | We currently use the following naming conventions for device names: | |
66 | ||
67 | * Ethernet devices: en*, systemd network interface names. This naming scheme is | |
68 | used for new {pve} installations since version 5.0. | |
69 | ||
70 | * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) This naming | |
71 | scheme is used for {pve} hosts which were installed before the 5.0 | |
72 | release. When upgrading to 5.0, the names are kept as-is. | |
73 | ||
74 | * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`) | |
75 | ||
76 | * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...) | |
77 | ||
78 | * VLANs: Simply add the VLAN number to the device name, | |
79 | separated by a period (`eno1.50`, `bond1.30`) | |
80 | ||
81 | This makes it easier to debug networks problems, because the device | |
82 | name implies the device type. | |
83 | ||
84 | Systemd Network Interface Names | |
85 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
86 | ||
87 | Systemd uses the two character prefix 'en' for Ethernet network | |
88 | devices. The next characters depends on the device driver and the fact | |
89 | which schema matches first. | |
90 | ||
91 | * o<index>[n<phys_port_name>|d<dev_port>] — devices on board | |
92 | ||
93 | * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id | |
94 | ||
95 | * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id | |
96 | ||
97 | * x<MAC> — device by MAC address | |
98 | ||
99 | The most common patterns are: | |
100 | ||
101 | * eno1 — is the first on board NIC | |
102 | ||
103 | * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1. | |
104 | ||
105 | For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names]. | |
106 | ||
107 | Choosing a network configuration | |
108 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
109 | ||
110 | Depending on your current network organization and your resources you can | |
111 | choose either a bridged, routed, or masquerading networking setup. | |
112 | ||
113 | {pve} server in a private LAN, using an external gateway to reach the internet | |
114 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
115 | ||
116 | The *Bridged* model makes the most sense in this case, and this is also | |
117 | the default mode on new {pve} installations. | |
118 | Each of your Guest system will have a virtual interface attached to the | |
119 | {pve} bridge. This is similar in effect to having the Guest network card | |
120 | directly connected to a new switch on your LAN, the {pve} host playing the role | |
121 | of the switch. | |
122 | ||
123 | {pve} server at hosting provider, with public IP ranges for Guests | |
124 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
125 | ||
126 | For this setup, you can use either a *Bridged* or *Routed* model, depending on | |
127 | what your provider allows. | |
128 | ||
129 | {pve} server at hosting provider, with a single public IP address | |
130 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
131 | ||
132 | In that case the only way to get outgoing network accesses for your guest | |
133 | systems is to use *Masquerading*. For incoming network access to your guests, | |
134 | you will need to configure *Port Forwarding*. | |
135 | ||
136 | For further flexibility, you can configure | |
137 | VLANs (IEEE 802.1q) and network bonding, also known as "link | |
138 | aggregation". That way it is possible to build complex and flexible | |
139 | virtual networks. | |
140 | ||
141 | Default Configuration using a Bridge | |
142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
143 | ||
144 | [thumbnail="default-network-setup-bridge.svg"] | |
145 | Bridges are like physical network switches implemented in software. | |
146 | All virtual guests can share a single bridge, or you can create multiple | |
147 | bridges to separate network domains. Each host can have up to 4094 bridges. | |
148 | ||
149 | The installation program creates a single bridge named `vmbr0`, which | |
150 | is connected to the first Ethernet card. The corresponding | |
151 | configuration in `/etc/network/interfaces` might look like this: | |
152 | ||
153 | ---- | |
154 | auto lo | |
155 | iface lo inet loopback | |
156 | ||
157 | iface eno1 inet manual | |
158 | ||
159 | auto vmbr0 | |
160 | iface vmbr0 inet static | |
161 | address 192.168.10.2 | |
162 | netmask 255.255.255.0 | |
163 | gateway 192.168.10.1 | |
164 | bridge-ports eno1 | |
165 | bridge-stp off | |
166 | bridge-fd 0 | |
167 | ---- | |
168 | ||
169 | Virtual machines behave as if they were directly connected to the | |
170 | physical network. The network, in turn, sees each virtual machine as | |
171 | having its own MAC, even though there is only one network cable | |
172 | connecting all of these VMs to the network. | |
173 | ||
174 | Routed Configuration | |
175 | ~~~~~~~~~~~~~~~~~~~~ | |
176 | ||
177 | Most hosting providers do not support the above setup. For security | |
178 | reasons, they disable networking as soon as they detect multiple MAC | |
179 | addresses on a single interface. | |
180 | ||
181 | TIP: Some providers allow you to register additional MACs through their | |
182 | management interface. This avoids the problem, but can be clumsy to | |
183 | configure because you need to register a MAC for each of your VMs. | |
184 | ||
185 | You can avoid the problem by ``routing'' all traffic via a single | |
186 | interface. This makes sure that all network packets use the same MAC | |
187 | address. | |
188 | ||
189 | [thumbnail="default-network-setup-routed.svg"] | |
190 | A common scenario is that you have a public IP (assume `198.51.100.5` | |
191 | for this example), and an additional IP block for your VMs | |
192 | (`203.0.113.16/29`). We recommend the following setup for such | |
193 | situations: | |
194 | ||
195 | ---- | |
196 | auto lo | |
197 | iface lo inet loopback | |
198 | ||
199 | auto eno1 | |
200 | iface eno1 inet static | |
201 | address 198.51.100.5 | |
202 | netmask 255.255.255.0 | |
203 | gateway 198.51.100.1 | |
204 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward | |
205 | post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp | |
206 | ||
207 | ||
208 | auto vmbr0 | |
209 | iface vmbr0 inet static | |
210 | address 203.0.113.17 | |
211 | netmask 255.255.255.248 | |
212 | bridge-ports none | |
213 | bridge-stp off | |
214 | bridge-fd 0 | |
215 | ---- | |
216 | ||
217 | ||
218 | Masquerading (NAT) with `iptables` | |
219 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
220 | ||
221 | Masquerading allows guests having only a private IP address to access the | |
222 | network by using the host IP address for outgoing traffic. Each outgoing | |
223 | packet is rewritten by `iptables` to appear as originating from the host, | |
224 | and responses are rewritten accordingly to be routed to the original sender. | |
225 | ||
226 | ---- | |
227 | auto lo | |
228 | iface lo inet loopback | |
229 | ||
230 | auto eno1 | |
231 | #real IP address | |
232 | iface eno1 inet static | |
233 | address 198.51.100.5 | |
234 | netmask 255.255.255.0 | |
235 | gateway 198.51.100.1 | |
236 | ||
237 | auto vmbr0 | |
238 | #private sub network | |
239 | iface vmbr0 inet static | |
240 | address 10.10.10.1 | |
241 | netmask 255.255.255.0 | |
242 | bridge-ports none | |
243 | bridge-stp off | |
244 | bridge-fd 0 | |
245 | ||
246 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward | |
247 | post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE | |
248 | post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE | |
249 | ---- | |
250 | ||
251 | NOTE: In some masquerade setups with firewall enabled, conntrack zones might be | |
252 | needed for outgoing connections. Otherwise the firewall could block outgoing | |
253 | connections since they will prefer the `POSTROUTING` of the VM bridge (and not | |
254 | `MASQUERADE`). | |
255 | ||
256 | Adding these lines in the `/etc/network/interfaces` can fix this problem: | |
257 | ||
258 | ---- | |
259 | post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 | |
260 | post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 | |
261 | ---- | |
262 | ||
263 | For more information about this, refer to the following links: | |
264 | ||
265 | https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg[Netfilter Packet Flow] | |
266 | ||
267 | https://lwn.net/Articles/370152/[Patch on netdev-list introducing conntrack zones] | |
268 | ||
269 | https://blog.lobraun.de/2019/05/19/prox/[Blog post with a good explanation by using TRACE in the raw table] | |
270 | ||
271 | ||
272 | ||
273 | Linux Bond | |
274 | ~~~~~~~~~~ | |
275 | ||
276 | Bonding (also called NIC teaming or Link Aggregation) is a technique | |
277 | for binding multiple NIC's to a single network device. It is possible | |
278 | to achieve different goals, like make the network fault-tolerant, | |
279 | increase the performance or both together. | |
280 | ||
281 | High-speed hardware like Fibre Channel and the associated switching | |
282 | hardware can be quite expensive. By doing link aggregation, two NICs | |
283 | can appear as one logical interface, resulting in double speed. This | |
284 | is a native Linux kernel feature that is supported by most | |
285 | switches. If your nodes have multiple Ethernet ports, you can | |
286 | distribute your points of failure by running network cables to | |
287 | different switches and the bonded connection will failover to one | |
288 | cable or the other in case of network trouble. | |
289 | ||
290 | Aggregated links can improve live-migration delays and improve the | |
291 | speed of replication of data between Proxmox VE Cluster nodes. | |
292 | ||
293 | There are 7 modes for bonding: | |
294 | ||
295 | * *Round-robin (balance-rr):* Transmit network packets in sequential | |
296 | order from the first available network interface (NIC) slave through | |
297 | the last. This mode provides load balancing and fault tolerance. | |
298 | ||
299 | * *Active-backup (active-backup):* Only one NIC slave in the bond is | |
300 | active. A different slave becomes active if, and only if, the active | |
301 | slave fails. The single logical bonded interface's MAC address is | |
302 | externally visible on only one NIC (port) to avoid distortion in the | |
303 | network switch. This mode provides fault tolerance. | |
304 | ||
305 | * *XOR (balance-xor):* Transmit network packets based on [(source MAC | |
306 | address XOR'd with destination MAC address) modulo NIC slave | |
307 | count]. This selects the same NIC slave for each destination MAC | |
308 | address. This mode provides load balancing and fault tolerance. | |
309 | ||
310 | * *Broadcast (broadcast):* Transmit network packets on all slave | |
311 | network interfaces. This mode provides fault tolerance. | |
312 | ||
313 | * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates | |
314 | aggregation groups that share the same speed and duplex | |
315 | settings. Utilizes all slave network interfaces in the active | |
316 | aggregator group according to the 802.3ad specification. | |
317 | ||
318 | * *Adaptive transmit load balancing (balance-tlb):* Linux bonding | |
319 | driver mode that does not require any special network-switch | |
320 | support. The outgoing network packet traffic is distributed according | |
321 | to the current load (computed relative to the speed) on each network | |
322 | interface slave. Incoming traffic is received by one currently | |
323 | designated slave network interface. If this receiving slave fails, | |
324 | another slave takes over the MAC address of the failed receiving | |
325 | slave. | |
326 | ||
327 | * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive | |
328 | load balancing (rlb) for IPV4 traffic, and does not require any | |
329 | special network switch support. The receive load balancing is achieved | |
330 | by ARP negotiation. The bonding driver intercepts the ARP Replies sent | |
331 | by the local system on their way out and overwrites the source | |
332 | hardware address with the unique hardware address of one of the NIC | |
333 | slaves in the single logical bonded interface such that different | |
334 | network-peers use different MAC addresses for their network packet | |
335 | traffic. | |
336 | ||
337 | If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using | |
338 | the corresponding bonding mode (802.3ad). Otherwise you should generally use the | |
339 | active-backup mode. + | |
340 | // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html | |
341 | If you intend to run your cluster network on the bonding interfaces, then you | |
342 | have to use active-passive mode on the bonding interfaces, other modes are | |
343 | unsupported. | |
344 | ||
345 | The following bond configuration can be used as distributed/shared | |
346 | storage network. The benefit would be that you get more speed and the | |
347 | network will be fault-tolerant. | |
348 | ||
349 | .Example: Use bond with fixed IP address | |
350 | ---- | |
351 | auto lo | |
352 | iface lo inet loopback | |
353 | ||
354 | iface eno1 inet manual | |
355 | ||
356 | iface eno2 inet manual | |
357 | ||
358 | iface eno3 inet manual | |
359 | ||
360 | auto bond0 | |
361 | iface bond0 inet static | |
362 | bond-slaves eno1 eno2 | |
363 | address 192.168.1.2 | |
364 | netmask 255.255.255.0 | |
365 | bond-miimon 100 | |
366 | bond-mode 802.3ad | |
367 | bond-xmit-hash-policy layer2+3 | |
368 | ||
369 | auto vmbr0 | |
370 | iface vmbr0 inet static | |
371 | address 10.10.10.2 | |
372 | netmask 255.255.255.0 | |
373 | gateway 10.10.10.1 | |
374 | bridge-ports eno3 | |
375 | bridge-stp off | |
376 | bridge-fd 0 | |
377 | ||
378 | ---- | |
379 | ||
380 | ||
381 | [thumbnail="default-network-setup-bond.svg"] | |
382 | Another possibility it to use the bond directly as bridge port. | |
383 | This can be used to make the guest network fault-tolerant. | |
384 | ||
385 | .Example: Use a bond as bridge port | |
386 | ---- | |
387 | auto lo | |
388 | iface lo inet loopback | |
389 | ||
390 | iface eno1 inet manual | |
391 | ||
392 | iface eno2 inet manual | |
393 | ||
394 | auto bond0 | |
395 | iface bond0 inet manual | |
396 | bond-slaves eno1 eno2 | |
397 | bond-miimon 100 | |
398 | bond-mode 802.3ad | |
399 | bond-xmit-hash-policy layer2+3 | |
400 | ||
401 | auto vmbr0 | |
402 | iface vmbr0 inet static | |
403 | address 10.10.10.2 | |
404 | netmask 255.255.255.0 | |
405 | gateway 10.10.10.1 | |
406 | bridge-ports bond0 | |
407 | bridge-stp off | |
408 | bridge-fd 0 | |
409 | ||
410 | ---- | |
411 | ||
412 | ||
413 | VLAN 802.1Q | |
414 | ~~~~~~~~~~~ | |
415 | ||
416 | A virtual LAN (VLAN) is a broadcast domain that is partitioned and | |
417 | isolated in the network at layer two. So it is possible to have | |
418 | multiple networks (4096) in a physical network, each independent of | |
419 | the other ones. | |
420 | ||
421 | Each VLAN network is identified by a number often called 'tag'. | |
422 | Network packages are then 'tagged' to identify which virtual network | |
423 | they belong to. | |
424 | ||
425 | ||
426 | VLAN for Guest Networks | |
427 | ^^^^^^^^^^^^^^^^^^^^^^^ | |
428 | ||
429 | {pve} supports this setup out of the box. You can specify the VLAN tag | |
430 | when you create a VM. The VLAN tag is part of the guest network | |
431 | configuration. The networking layer supports different modes to | |
432 | implement VLANs, depending on the bridge configuration: | |
433 | ||
434 | * *VLAN awareness on the Linux bridge:* | |
435 | In this case, each guest's virtual network card is assigned to a VLAN tag, | |
436 | which is transparently supported by the Linux bridge. | |
437 | Trunk mode is also possible, but that makes configuration | |
438 | in the guest necessary. | |
439 | ||
440 | * *"traditional" VLAN on the Linux bridge:* | |
441 | In contrast to the VLAN awareness method, this method is not transparent | |
442 | and creates a VLAN device with associated bridge for each VLAN. | |
443 | That is, creating a guest on VLAN 5 for example, would create two | |
444 | interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs. | |
445 | ||
446 | * *Open vSwitch VLAN:* | |
447 | This mode uses the OVS VLAN feature. | |
448 | ||
449 | * *Guest configured VLAN:* | |
450 | VLANs are assigned inside the guest. In this case, the setup is | |
451 | completely done inside the guest and can not be influenced from the | |
452 | outside. The benefit is that you can use more than one VLAN on a | |
453 | single virtual NIC. | |
454 | ||
455 | ||
456 | VLAN on the Host | |
457 | ^^^^^^^^^^^^^^^^ | |
458 | ||
459 | To allow host communication with an isolated network. It is possible | |
460 | to apply VLAN tags to any network device (NIC, Bond, Bridge). In | |
461 | general, you should configure the VLAN on the interface with the least | |
462 | abstraction layers between itself and the physical NIC. | |
463 | ||
464 | For example, in a default configuration where you want to place | |
465 | the host management address on a separate VLAN. | |
466 | ||
467 | ||
468 | .Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge | |
469 | ---- | |
470 | auto lo | |
471 | iface lo inet loopback | |
472 | ||
473 | iface eno1 inet manual | |
474 | ||
475 | iface eno1.5 inet manual | |
476 | ||
477 | auto vmbr0v5 | |
478 | iface vmbr0v5 inet static | |
479 | address 10.10.10.2 | |
480 | netmask 255.255.255.0 | |
481 | gateway 10.10.10.1 | |
482 | bridge-ports eno1.5 | |
483 | bridge-stp off | |
484 | bridge-fd 0 | |
485 | ||
486 | auto vmbr0 | |
487 | iface vmbr0 inet manual | |
488 | bridge-ports eno1 | |
489 | bridge-stp off | |
490 | bridge-fd 0 | |
491 | ||
492 | ---- | |
493 | ||
494 | .Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge | |
495 | ---- | |
496 | auto lo | |
497 | iface lo inet loopback | |
498 | ||
499 | iface eno1 inet manual | |
500 | ||
501 | ||
502 | auto vmbr0.5 | |
503 | iface vmbr0.5 inet static | |
504 | address 10.10.10.2 | |
505 | netmask 255.255.255.0 | |
506 | gateway 10.10.10.1 | |
507 | ||
508 | auto vmbr0 | |
509 | iface vmbr0 inet manual | |
510 | bridge-ports eno1 | |
511 | bridge-stp off | |
512 | bridge-fd 0 | |
513 | bridge-vlan-aware yes | |
514 | ---- | |
515 | ||
516 | The next example is the same setup but a bond is used to | |
517 | make this network fail-safe. | |
518 | ||
519 | .Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge | |
520 | ---- | |
521 | auto lo | |
522 | iface lo inet loopback | |
523 | ||
524 | iface eno1 inet manual | |
525 | ||
526 | iface eno2 inet manual | |
527 | ||
528 | auto bond0 | |
529 | iface bond0 inet manual | |
530 | bond-slaves eno1 eno2 | |
531 | bond-miimon 100 | |
532 | bond-mode 802.3ad | |
533 | bond-xmit-hash-policy layer2+3 | |
534 | ||
535 | iface bond0.5 inet manual | |
536 | ||
537 | auto vmbr0v5 | |
538 | iface vmbr0v5 inet static | |
539 | address 10.10.10.2 | |
540 | netmask 255.255.255.0 | |
541 | gateway 10.10.10.1 | |
542 | bridge-ports bond0.5 | |
543 | bridge-stp off | |
544 | bridge-fd 0 | |
545 | ||
546 | auto vmbr0 | |
547 | iface vmbr0 inet manual | |
548 | bridge-ports bond0 | |
549 | bridge-stp off | |
550 | bridge-fd 0 | |
551 | ||
552 | ---- | |
553 | ||
554 | //// | |
555 | TODO: explain IPv6 support? | |
556 | TODO: explain OVS | |
557 | //// |