1 [[sysadmin_network_configuration]]
8 {pve} is using the Linux network stack. This provides a lot of flexibility on
9 how to set up the network on the {pve} nodes. The configuration can be done
10 either via the GUI, or by manually editing the file `/etc/network/interfaces`,
11 which contains the whole network configuration. The `interfaces(5)` manual
12 page contains the complete format description. All {pve} tools try hard to keep
13 direct user modifications, but using the GUI is still preferable, because it
14 protects you from errors.
16 A Linux bridge interface (commonly called 'vmbrX') is needed to connect guests
17 to the underlying physical network. It can be thought of as a virtual switch
18 which the guests and physical interfaces are connected to. This section provides
19 some examples on how the network can be set up to accomodate different use cases
20 like redundancy with a xref:sysadmin_network_bond['bond'],
21 xref:sysadmin_network_vlan['vlans'] or
22 xref:sysadmin_network_routed['routed'] and
23 xref:sysadmin_network_masquerading['NAT'] setups.
25 The xref:chapter_pvesdn[Software Defined Network] is an option for more complex
26 virtual networks in {pve} clusters.
28 WARNING: It's discouraged to use the traditional Debian tools `ifup` and `ifdown`
29 if unsure, as they have some pitfalls like interupting all guest traffic on
30 `ifdown vmbrX` but not reconnecting those guest again when doing `ifup` on the
36 {pve} does not write changes directly to `/etc/network/interfaces`. Instead, we
37 write into a temporary file called `/etc/network/interfaces.new`, this way you
38 can do many related changes at once. This also allows to ensure your changes
39 are correct before applying, as a wrong network configuration may render a node
42 Live-Reload Network with ifupdown2
43 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
45 With the recommended 'ifupdown2' package (default for new installations since
46 {pve} 7.0), it is possible to apply network configuration changes without a
47 reboot. If you change the network configuration via the GUI, you can click the
48 'Apply Configuration' button. This will move changes from the staging
49 `interfaces.new` file to `/etc/network/interfaces` and apply them live.
51 If you made manual changes directly to the `/etc/network/interfaces` file, you
52 can apply them by running `ifreload -a`
54 NOTE: If you installed {pve} on top of Debian, or upgraded to {pve} 7.0 from an
55 older {pve} installation, make sure 'ifupdown2' is installed: `apt install
61 Another way to apply a new network configuration is to reboot the node.
62 In that case the systemd service `pvenetcommit` will activate the staging
63 `interfaces.new` file before the `networking` service will apply that
69 We currently use the following naming conventions for device names:
71 * Ethernet devices: `en*`, systemd network interface names. This naming scheme is
72 used for new {pve} installations since version 5.0.
74 * Ethernet devices: `eth[N]`, where 0 ≤ N (`eth0`, `eth1`, ...) This naming
75 scheme is used for {pve} hosts which were installed before the 5.0
76 release. When upgrading to 5.0, the names are kept as-is.
78 * Bridge names: Commonly `vmbr[N]`, where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`),
79 but you can use any alphanumeric string that starts with a character and is at
80 most 10 characters long.
82 * Bonds: `bond[N]`, where 0 ≤ N (`bond0`, `bond1`, ...)
84 * VLANs: Simply add the VLAN number to the device name,
85 separated by a period (`eno1.50`, `bond1.30`)
87 This makes it easier to debug networks problems, because the device
88 name implies the device type.
90 [[systemd_network_interface_names]]
91 Systemd Network Interface Names
92 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
94 Systemd defines a versioned naming scheme for network device names. The
95 scheme uses the two-character prefix `en` for Ethernet network devices. The
96 next characters depends on the device driver, device location and other
97 attributes. Some possible patterns are:
99 * `o<index>[n<phys_port_name>|d<dev_port>]` — devices on board
101 * `s<slot>[f<function>][n<phys_port_name>|d<dev_port>]` — devices by hotplug id
103 * `[P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>]` —
106 * `x<MAC>` — devices by MAC address
108 Some examples for the most common patterns are:
110 * `eno1` — is the first on-board NIC
112 * `enp3s0f1` — is function 1 of the NIC on PCI bus 3, slot 0
114 For a full list of possible device name patterns, see the
115 https://manpages.debian.org/stable/systemd/systemd.net-naming-scheme.7.en.html[
116 systemd.net-naming-scheme(7) manpage].
118 A new version of systemd may define a new version of the network device naming
119 scheme, which it then uses by default. Consequently, updating to a newer
120 systemd version, for example during a major {pve} upgrade, can change the names
121 of network devices and require adjusting the network configuration. To avoid
122 name changes due to a new version of the naming scheme, you can manually pin a
123 particular naming scheme version (see
124 xref:network_pin_naming_scheme_version[below]).
126 However, even with a pinned naming scheme version, network device names can
127 still change due to kernel or driver updates. In order to avoid name changes
128 for a particular network device altogether, you can manually override its name
129 using a link file (see xref:network_override_device_names[below]).
131 For more information on network interface names, see
132 https://systemd.io/PREDICTABLE_INTERFACE_NAMES/[Predictable Network Interface
135 [[network_pin_naming_scheme_version]]
136 Pinning a specific naming scheme version
137 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
139 You can pin a specific version of the naming scheme for network devices by
140 adding the `net.naming-scheme=<version>` parameter to the
141 xref:sysboot_edit_kernel_cmdline[kernel command line]. For a list of naming
142 scheme versions, see the
143 https://manpages.debian.org/stable/systemd/systemd.net-naming-scheme.7.en.html[
144 systemd.net-naming-scheme(7) manpage].
146 For example, to pin the version `v252`, which is the latest naming scheme
147 version for a fresh {pve} 8.0 installation, add the following kernel
148 command-line parameter:
151 net.naming-scheme=v252
154 See also xref:sysboot_edit_kernel_cmdline[this section] on editing the kernel
155 command line. You need to reboot for the changes to take effect.
157 [[network_override_device_names]]
158 Overriding network device names
159 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
161 You can manually assign a name to a particular network device using a custom
162 https://manpages.debian.org/stable/udev/systemd.link.5.en.html[systemd.link
163 file]. This overrides the name that would be assigned according to the latest
164 network device naming scheme. This way, you can avoid naming changes due to
165 kernel updates, driver updates or newer versions of the naming scheme.
167 Custom link files should be placed in `/etc/systemd/network/` and named
168 `<n>-<id>.link`, where `n` is a priority smaller than `99` and `id` is some
169 identifier. A link file has two sections: `[Match]` determines which interfaces
170 the file will apply to; `[Link]` determines how these interfaces should be
171 configured, including their naming.
173 To assign a name to a particular network device, you need a way to uniquely and
174 permanently identify that device in the `[Match]` section. One possibility is
175 to match the device's MAC address using the `MACAddress` option, as it is
178 The `[Match]` section should also contain a `Type` option to make sure it only
179 matches the expected physical interface, and not bridge/bond/VLAN interfaces
180 with the same MAC address. In most setups, `Type` should be set to `ether` to
181 match only Ethernet devices, but some setups may require other choices. See the
182 https://manpages.debian.org/stable/udev/systemd.link.5.en.html[systemd.link(5)
183 manpage] for more details.
185 Then, you can assign a name using the `Name` option in the `[Link]` section.
187 Link files are copied to the `initramfs`, so it is recommended to refresh the
188 `initramfs` after adding, modifying, or removing a link file:
191 # update-initramfs -u -k all
194 For example, to assign the name `enwan0` to the Ethernet device with MAC
195 address `aa:bb:cc:dd:ee:ff`, create a file
196 `/etc/systemd/network/10-enwan0.link` with the following contents:
200 MACAddress=aa:bb:cc:dd:ee:ff
207 Do not forget to adjust `/etc/network/interfaces` to use the new name, and
208 refresh your `initramfs` as described above. You need to reboot the node for
209 the change to take effect.
211 NOTE: It is recommended to assign a name starting with `en` or `eth` so that
212 {pve} recognizes the interface as a physical network device which can then be
213 configured via the GUI. Also, you should ensure that the name will not clash
214 with other interface names in the future. One possibility is to assign a name
215 that does not match any name pattern that systemd uses for network interfaces
216 (xref:systemd_network_interface_names[see above]), such as `enwan0` in the
219 For more information on link files, see the
220 https://manpages.debian.org/stable/udev/systemd.link.5.en.html[systemd.link(5)
223 Choosing a network configuration
224 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
226 Depending on your current network organization and your resources you can
227 choose either a bridged, routed, or masquerading networking setup.
229 {pve} server in a private LAN, using an external gateway to reach the internet
230 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
232 The *Bridged* model makes the most sense in this case, and this is also
233 the default mode on new {pve} installations.
234 Each of your Guest system will have a virtual interface attached to the
235 {pve} bridge. This is similar in effect to having the Guest network card
236 directly connected to a new switch on your LAN, the {pve} host playing the role
239 {pve} server at hosting provider, with public IP ranges for Guests
240 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
242 For this setup, you can use either a *Bridged* or *Routed* model, depending on
243 what your provider allows.
245 {pve} server at hosting provider, with a single public IP address
246 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
248 In that case the only way to get outgoing network accesses for your guest
249 systems is to use *Masquerading*. For incoming network access to your guests,
250 you will need to configure *Port Forwarding*.
252 For further flexibility, you can configure
253 VLANs (IEEE 802.1q) and network bonding, also known as "link
254 aggregation". That way it is possible to build complex and flexible
257 Default Configuration using a Bridge
258 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
260 [thumbnail="default-network-setup-bridge.svg"]
261 Bridges are like physical network switches implemented in software.
262 All virtual guests can share a single bridge, or you can create multiple
263 bridges to separate network domains. Each host can have up to 4094 bridges.
265 The installation program creates a single bridge named `vmbr0`, which
266 is connected to the first Ethernet card. The corresponding
267 configuration in `/etc/network/interfaces` might look like this:
271 iface lo inet loopback
273 iface eno1 inet manual
276 iface vmbr0 inet static
277 address 192.168.10.2/24
284 Virtual machines behave as if they were directly connected to the
285 physical network. The network, in turn, sees each virtual machine as
286 having its own MAC, even though there is only one network cable
287 connecting all of these VMs to the network.
289 [[sysadmin_network_routed]]
293 Most hosting providers do not support the above setup. For security
294 reasons, they disable networking as soon as they detect multiple MAC
295 addresses on a single interface.
297 TIP: Some providers allow you to register additional MACs through their
298 management interface. This avoids the problem, but can be clumsy to
299 configure because you need to register a MAC for each of your VMs.
301 You can avoid the problem by ``routing'' all traffic via a single
302 interface. This makes sure that all network packets use the same MAC
305 [thumbnail="default-network-setup-routed.svg"]
306 A common scenario is that you have a public IP (assume `198.51.100.5`
307 for this example), and an additional IP block for your VMs
308 (`203.0.113.16/28`). We recommend the following setup for such
313 iface lo inet loopback
316 iface eno0 inet static
317 address 198.51.100.5/29
319 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
320 post-up echo 1 > /proc/sys/net/ipv4/conf/eno0/proxy_arp
324 iface vmbr0 inet static
325 address 203.0.113.17/28
332 [[sysadmin_network_masquerading]]
333 Masquerading (NAT) with `iptables`
334 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
336 Masquerading allows guests having only a private IP address to access the
337 network by using the host IP address for outgoing traffic. Each outgoing
338 packet is rewritten by `iptables` to appear as originating from the host,
339 and responses are rewritten accordingly to be routed to the original sender.
343 iface lo inet loopback
347 iface eno1 inet static
348 address 198.51.100.5/24
353 iface vmbr0 inet static
354 address 10.10.10.1/24
359 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
360 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
361 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
364 NOTE: In some masquerade setups with firewall enabled, conntrack zones might be
365 needed for outgoing connections. Otherwise the firewall could block outgoing
366 connections since they will prefer the `POSTROUTING` of the VM bridge (and not
369 Adding these lines in the `/etc/network/interfaces` can fix this problem:
372 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
373 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
376 For more information about this, refer to the following links:
378 https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg[Netfilter Packet Flow]
380 https://lwn.net/Articles/370152/[Patch on netdev-list introducing conntrack zones]
382 https://web.archive.org/web/20220610151210/https://blog.lobraun.de/2019/05/19/prox/[Blog post with a good explanation by using TRACE in the raw table]
385 [[sysadmin_network_bond]]
389 Bonding (also called NIC teaming or Link Aggregation) is a technique
390 for binding multiple NIC's to a single network device. It is possible
391 to achieve different goals, like make the network fault-tolerant,
392 increase the performance or both together.
394 High-speed hardware like Fibre Channel and the associated switching
395 hardware can be quite expensive. By doing link aggregation, two NICs
396 can appear as one logical interface, resulting in double speed. This
397 is a native Linux kernel feature that is supported by most
398 switches. If your nodes have multiple Ethernet ports, you can
399 distribute your points of failure by running network cables to
400 different switches and the bonded connection will failover to one
401 cable or the other in case of network trouble.
403 Aggregated links can improve live-migration delays and improve the
404 speed of replication of data between Proxmox VE Cluster nodes.
406 There are 7 modes for bonding:
408 * *Round-robin (balance-rr):* Transmit network packets in sequential
409 order from the first available network interface (NIC) slave through
410 the last. This mode provides load balancing and fault tolerance.
412 * *Active-backup (active-backup):* Only one NIC slave in the bond is
413 active. A different slave becomes active if, and only if, the active
414 slave fails. The single logical bonded interface's MAC address is
415 externally visible on only one NIC (port) to avoid distortion in the
416 network switch. This mode provides fault tolerance.
418 * *XOR (balance-xor):* Transmit network packets based on [(source MAC
419 address XOR'd with destination MAC address) modulo NIC slave
420 count]. This selects the same NIC slave for each destination MAC
421 address. This mode provides load balancing and fault tolerance.
423 * *Broadcast (broadcast):* Transmit network packets on all slave
424 network interfaces. This mode provides fault tolerance.
426 * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
427 aggregation groups that share the same speed and duplex
428 settings. Utilizes all slave network interfaces in the active
429 aggregator group according to the 802.3ad specification.
431 * *Adaptive transmit load balancing (balance-tlb):* Linux bonding
432 driver mode that does not require any special network-switch
433 support. The outgoing network packet traffic is distributed according
434 to the current load (computed relative to the speed) on each network
435 interface slave. Incoming traffic is received by one currently
436 designated slave network interface. If this receiving slave fails,
437 another slave takes over the MAC address of the failed receiving
440 * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive
441 load balancing (rlb) for IPV4 traffic, and does not require any
442 special network switch support. The receive load balancing is achieved
443 by ARP negotiation. The bonding driver intercepts the ARP Replies sent
444 by the local system on their way out and overwrites the source
445 hardware address with the unique hardware address of one of the NIC
446 slaves in the single logical bonded interface such that different
447 network-peers use different MAC addresses for their network packet
450 If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
451 the corresponding bonding mode (802.3ad). Otherwise you should generally use the
454 For the cluster network (Corosync) we recommend configuring it with multiple
455 networks. Corosync does not need a bond for network reduncancy as it can switch
456 between networks by itself, if one becomes unusable.
458 The following bond configuration can be used as distributed/shared
459 storage network. The benefit would be that you get more speed and the
460 network will be fault-tolerant.
462 .Example: Use bond with fixed IP address
465 iface lo inet loopback
467 iface eno1 inet manual
469 iface eno2 inet manual
471 iface eno3 inet manual
474 iface bond0 inet static
475 bond-slaves eno1 eno2
476 address 192.168.1.2/24
479 bond-xmit-hash-policy layer2+3
482 iface vmbr0 inet static
483 address 10.10.10.2/24
492 [thumbnail="default-network-setup-bond.svg"]
493 Another possibility it to use the bond directly as bridge port.
494 This can be used to make the guest network fault-tolerant.
496 .Example: Use a bond as bridge port
499 iface lo inet loopback
501 iface eno1 inet manual
503 iface eno2 inet manual
506 iface bond0 inet manual
507 bond-slaves eno1 eno2
510 bond-xmit-hash-policy layer2+3
513 iface vmbr0 inet static
514 address 10.10.10.2/24
523 [[sysadmin_network_vlan]]
527 A virtual LAN (VLAN) is a broadcast domain that is partitioned and
528 isolated in the network at layer two. So it is possible to have
529 multiple networks (4096) in a physical network, each independent of
532 Each VLAN network is identified by a number often called 'tag'.
533 Network packages are then 'tagged' to identify which virtual network
537 VLAN for Guest Networks
538 ^^^^^^^^^^^^^^^^^^^^^^^
540 {pve} supports this setup out of the box. You can specify the VLAN tag
541 when you create a VM. The VLAN tag is part of the guest network
542 configuration. The networking layer supports different modes to
543 implement VLANs, depending on the bridge configuration:
545 * *VLAN awareness on the Linux bridge:*
546 In this case, each guest's virtual network card is assigned to a VLAN tag,
547 which is transparently supported by the Linux bridge.
548 Trunk mode is also possible, but that makes configuration
549 in the guest necessary.
551 * *"traditional" VLAN on the Linux bridge:*
552 In contrast to the VLAN awareness method, this method is not transparent
553 and creates a VLAN device with associated bridge for each VLAN.
554 That is, creating a guest on VLAN 5 for example, would create two
555 interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs.
557 * *Open vSwitch VLAN:*
558 This mode uses the OVS VLAN feature.
560 * *Guest configured VLAN:*
561 VLANs are assigned inside the guest. In this case, the setup is
562 completely done inside the guest and can not be influenced from the
563 outside. The benefit is that you can use more than one VLAN on a
570 To allow host communication with an isolated network. It is possible
571 to apply VLAN tags to any network device (NIC, Bond, Bridge). In
572 general, you should configure the VLAN on the interface with the least
573 abstraction layers between itself and the physical NIC.
575 For example, in a default configuration where you want to place
576 the host management address on a separate VLAN.
579 .Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge
582 iface lo inet loopback
584 iface eno1 inet manual
586 iface eno1.5 inet manual
589 iface vmbr0v5 inet static
590 address 10.10.10.2/24
597 iface vmbr0 inet manual
604 .Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
607 iface lo inet loopback
609 iface eno1 inet manual
613 iface vmbr0.5 inet static
614 address 10.10.10.2/24
618 iface vmbr0 inet manual
622 bridge-vlan-aware yes
626 The next example is the same setup but a bond is used to
627 make this network fail-safe.
629 .Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge
632 iface lo inet loopback
634 iface eno1 inet manual
636 iface eno2 inet manual
639 iface bond0 inet manual
640 bond-slaves eno1 eno2
643 bond-xmit-hash-policy layer2+3
645 iface bond0.5 inet manual
648 iface vmbr0v5 inet static
649 address 10.10.10.2/24
656 iface vmbr0 inet manual
663 Disabling IPv6 on the Node
664 ~~~~~~~~~~~~~~~~~~~~~~~~~~
666 {pve} works correctly in all environments, irrespective of whether IPv6 is
667 deployed or not. We recommend leaving all settings at the provided defaults.
669 Should you still need to disable support for IPv6 on your node, do so by
670 creating an appropriate `sysctl.conf (5)` snippet file and setting the proper
671 https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt[sysctls],
672 for example adding `/etc/sysctl.d/disable-ipv6.conf` with content:
675 net.ipv6.conf.all.disable_ipv6 = 1
676 net.ipv6.conf.default.disable_ipv6 = 1
679 This method is preferred to disabling the loading of the IPv6 module on the
680 https://www.kernel.org/doc/Documentation/networking/ipv6.rst[kernel commandline].
683 Disabling MAC Learning on a Bridge
684 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
686 By default, MAC learning is enabled on a bridge to ensure a smooth experience
687 with virtual guests and their networks.
689 But in some environments this can be undesired. Since {pve} 7.3 you can disable
690 MAC learning on the bridge by setting the `bridge-disable-mac-learning 1`
691 configuration on a bridge in `/etc/network/interfaces', for example:
697 iface vmbr0 inet static
698 address 10.10.10.2/24
703 bridge-disable-mac-learning 1
706 Once enabled, {pve} will manually add the configured MAC address from VMs and
707 Containers to the bridges forwarding database to ensure that guest can still
708 use the network - but only when they are using their actual MAC address.
711 TODO: explain IPv6 support?