1 [[sysadmin_network_configuration]]
8 Network configuration can be done either via the GUI, or by manually
9 editing the file `/etc/network/interfaces`, which contains the
10 whole network configuration. The `interfaces(5)` manual page contains the
11 complete format description. All {pve} tools try hard to keep direct
12 user modifications, but using the GUI is still preferable, because it
13 protects you from errors.
15 Once the network is configured, you can use the Debian traditional tools `ifup`
16 and `ifdown` commands to bring interfaces up and down.
21 {pve} does not write changes directly to `/etc/network/interfaces`. Instead, we
22 write into a temporary file called `/etc/network/interfaces.new`, this way you
23 can do many related changes at once. This also allows to ensure your changes
24 are correct before applying, as a wrong network configuration may render a node
30 With the default installed `ifupdown` network managing package you need to
31 reboot to commit any pending network changes. Most of the time, the basic {pve}
32 network setup is stable and does not change often, so rebooting should not be
35 Reload Network with ifupdown2
36 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38 With the optional `ifupdown2` network managing package you also can reload the
39 network configuration live, without requiring a reboot.
41 Since {pve} 6.1 you can apply pending network changes over the web-interface,
42 using the 'Apply Configuration' button in the 'Network' panel of a node.
44 To install 'ifupdown2' ensure you have the latest {pve} updates installed, then
46 WARNING: installing 'ifupdown2' will remove 'ifupdown', but as the removal
47 scripts of 'ifupdown' before version '0.8.35+pve1' have a issue where network
48 is fully stopped on removal footnote:[Introduced with Debian Buster:
49 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945877] you *must* ensure
50 that you have a up to date 'ifupdown' package version.
52 For the installation itself you can then simply do:
56 With that you're all set. You can also switch back to the 'ifupdown' variant at
57 any time, if you run into issues.
62 We currently use the following naming conventions for device names:
64 * Ethernet devices: en*, systemd network interface names. This naming scheme is
65 used for new {pve} installations since version 5.0.
67 * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) This naming
68 scheme is used for {pve} hosts which were installed before the 5.0
69 release. When upgrading to 5.0, the names are kept as-is.
71 * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`)
73 * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...)
75 * VLANs: Simply add the VLAN number to the device name,
76 separated by a period (`eno1.50`, `bond1.30`)
78 This makes it easier to debug networks problems, because the device
79 name implies the device type.
81 Systemd Network Interface Names
82 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
84 Systemd uses the two character prefix 'en' for Ethernet network
85 devices. The next characters depends on the device driver and the fact
86 which schema matches first.
88 * o<index>[n<phys_port_name>|d<dev_port>] — devices on board
90 * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id
92 * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id
94 * x<MAC> — device by MAC address
96 The most common patterns are:
98 * eno1 — is the first on board NIC
100 * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.
102 For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names].
104 Choosing a network configuration
105 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107 Depending on your current network organization and your resources you can
108 choose either a bridged, routed, or masquerading networking setup.
110 {pve} server in a private LAN, using an external gateway to reach the internet
111 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
113 The *Bridged* model makes the most sense in this case, and this is also
114 the default mode on new {pve} installations.
115 Each of your Guest system will have a virtual interface attached to the
116 {pve} bridge. This is similar in effect to having the Guest network card
117 directly connected to a new switch on your LAN, the {pve} host playing the role
120 {pve} server at hosting provider, with public IP ranges for Guests
121 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
123 For this setup, you can use either a *Bridged* or *Routed* model, depending on
124 what your provider allows.
126 {pve} server at hosting provider, with a single public IP address
127 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
129 In that case the only way to get outgoing network accesses for your guest
130 systems is to use *Masquerading*. For incoming network access to your guests,
131 you will need to configure *Port Forwarding*.
133 For further flexibility, you can configure
134 VLANs (IEEE 802.1q) and network bonding, also known as "link
135 aggregation". That way it is possible to build complex and flexible
138 Default Configuration using a Bridge
139 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
141 [thumbnail="default-network-setup-bridge.svg"]
142 Bridges are like physical network switches implemented in software.
143 All virtual guests can share a single bridge, or you can create multiple
144 bridges to separate network domains. Each host can have up to 4094 bridges.
146 The installation program creates a single bridge named `vmbr0`, which
147 is connected to the first Ethernet card. The corresponding
148 configuration in `/etc/network/interfaces` might look like this:
152 iface lo inet loopback
154 iface eno1 inet manual
157 iface vmbr0 inet static
159 netmask 255.255.255.0
166 Virtual machines behave as if they were directly connected to the
167 physical network. The network, in turn, sees each virtual machine as
168 having its own MAC, even though there is only one network cable
169 connecting all of these VMs to the network.
174 Most hosting providers do not support the above setup. For security
175 reasons, they disable networking as soon as they detect multiple MAC
176 addresses on a single interface.
178 TIP: Some providers allow you to register additional MACs through their
179 management interface. This avoids the problem, but can be clumsy to
180 configure because you need to register a MAC for each of your VMs.
182 You can avoid the problem by ``routing'' all traffic via a single
183 interface. This makes sure that all network packets use the same MAC
186 [thumbnail="default-network-setup-routed.svg"]
187 A common scenario is that you have a public IP (assume `198.51.100.5`
188 for this example), and an additional IP block for your VMs
189 (`203.0.113.16/29`). We recommend the following setup for such
194 iface lo inet loopback
197 iface eno1 inet static
199 netmask 255.255.255.0
201 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
202 post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
206 iface vmbr0 inet static
208 netmask 255.255.255.248
215 Masquerading (NAT) with `iptables`
216 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
218 Masquerading allows guests having only a private IP address to access the
219 network by using the host IP address for outgoing traffic. Each outgoing
220 packet is rewritten by `iptables` to appear as originating from the host,
221 and responses are rewritten accordingly to be routed to the original sender.
225 iface lo inet loopback
229 iface eno1 inet static
231 netmask 255.255.255.0
236 iface vmbr0 inet static
238 netmask 255.255.255.0
243 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
244 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
245 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
248 NOTE: In some masquerade setups with firewall enabled, conntrack zones might be
249 needed for outgoing connections. Otherwise the firewall could block outgoing
250 connections since they will prefer the `POSTROUTING` of the VM bridge (and not
253 Adding these lines in the `/etc/network/interfaces` can fix this problem:
256 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
257 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
260 For more information about this, refer to the following links:
262 https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg[Netfilter Packet Flow]
264 https://lwn.net/Articles/370152/[Patch on netdev-list introducing conntrack zones]
266 https://blog.lobraun.de/2019/05/19/prox/[Blog post with a good explanation by using TRACE in the raw table]
273 Bonding (also called NIC teaming or Link Aggregation) is a technique
274 for binding multiple NIC's to a single network device. It is possible
275 to achieve different goals, like make the network fault-tolerant,
276 increase the performance or both together.
278 High-speed hardware like Fibre Channel and the associated switching
279 hardware can be quite expensive. By doing link aggregation, two NICs
280 can appear as one logical interface, resulting in double speed. This
281 is a native Linux kernel feature that is supported by most
282 switches. If your nodes have multiple Ethernet ports, you can
283 distribute your points of failure by running network cables to
284 different switches and the bonded connection will failover to one
285 cable or the other in case of network trouble.
287 Aggregated links can improve live-migration delays and improve the
288 speed of replication of data between Proxmox VE Cluster nodes.
290 There are 7 modes for bonding:
292 * *Round-robin (balance-rr):* Transmit network packets in sequential
293 order from the first available network interface (NIC) slave through
294 the last. This mode provides load balancing and fault tolerance.
296 * *Active-backup (active-backup):* Only one NIC slave in the bond is
297 active. A different slave becomes active if, and only if, the active
298 slave fails. The single logical bonded interface's MAC address is
299 externally visible on only one NIC (port) to avoid distortion in the
300 network switch. This mode provides fault tolerance.
302 * *XOR (balance-xor):* Transmit network packets based on [(source MAC
303 address XOR'd with destination MAC address) modulo NIC slave
304 count]. This selects the same NIC slave for each destination MAC
305 address. This mode provides load balancing and fault tolerance.
307 * *Broadcast (broadcast):* Transmit network packets on all slave
308 network interfaces. This mode provides fault tolerance.
310 * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
311 aggregation groups that share the same speed and duplex
312 settings. Utilizes all slave network interfaces in the active
313 aggregator group according to the 802.3ad specification.
315 * *Adaptive transmit load balancing (balance-tlb):* Linux bonding
316 driver mode that does not require any special network-switch
317 support. The outgoing network packet traffic is distributed according
318 to the current load (computed relative to the speed) on each network
319 interface slave. Incoming traffic is received by one currently
320 designated slave network interface. If this receiving slave fails,
321 another slave takes over the MAC address of the failed receiving
324 * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive
325 load balancing (rlb) for IPV4 traffic, and does not require any
326 special network switch support. The receive load balancing is achieved
327 by ARP negotiation. The bonding driver intercepts the ARP Replies sent
328 by the local system on their way out and overwrites the source
329 hardware address with the unique hardware address of one of the NIC
330 slaves in the single logical bonded interface such that different
331 network-peers use different MAC addresses for their network packet
334 If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
335 the corresponding bonding mode (802.3ad). Otherwise you should generally use the
336 active-backup mode. +
337 // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html
338 If you intend to run your cluster network on the bonding interfaces, then you
339 have to use active-passive mode on the bonding interfaces, other modes are
342 The following bond configuration can be used as distributed/shared
343 storage network. The benefit would be that you get more speed and the
344 network will be fault-tolerant.
346 .Example: Use bond with fixed IP address
349 iface lo inet loopback
351 iface eno1 inet manual
353 iface eno2 inet manual
355 iface eno3 inet manual
358 iface bond0 inet static
359 bond-slaves eno1 eno2
361 netmask 255.255.255.0
364 bond-xmit-hash-policy layer2+3
367 iface vmbr0 inet static
369 netmask 255.255.255.0
378 [thumbnail="default-network-setup-bond.svg"]
379 Another possibility it to use the bond directly as bridge port.
380 This can be used to make the guest network fault-tolerant.
382 .Example: Use a bond as bridge port
385 iface lo inet loopback
387 iface eno1 inet manual
389 iface eno2 inet manual
392 iface bond0 inet manual
393 bond-slaves eno1 eno2
396 bond-xmit-hash-policy layer2+3
399 iface vmbr0 inet static
401 netmask 255.255.255.0
413 A virtual LAN (VLAN) is a broadcast domain that is partitioned and
414 isolated in the network at layer two. So it is possible to have
415 multiple networks (4096) in a physical network, each independent of
418 Each VLAN network is identified by a number often called 'tag'.
419 Network packages are then 'tagged' to identify which virtual network
423 VLAN for Guest Networks
424 ^^^^^^^^^^^^^^^^^^^^^^^
426 {pve} supports this setup out of the box. You can specify the VLAN tag
427 when you create a VM. The VLAN tag is part of the guest network
428 configuration. The networking layer supports different modes to
429 implement VLANs, depending on the bridge configuration:
431 * *VLAN awareness on the Linux bridge:*
432 In this case, each guest's virtual network card is assigned to a VLAN tag,
433 which is transparently supported by the Linux bridge.
434 Trunk mode is also possible, but that makes configuration
435 in the guest necessary.
437 * *"traditional" VLAN on the Linux bridge:*
438 In contrast to the VLAN awareness method, this method is not transparent
439 and creates a VLAN device with associated bridge for each VLAN.
440 That is, creating a guest on VLAN 5 for example, would create two
441 interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs.
443 * *Open vSwitch VLAN:*
444 This mode uses the OVS VLAN feature.
446 * *Guest configured VLAN:*
447 VLANs are assigned inside the guest. In this case, the setup is
448 completely done inside the guest and can not be influenced from the
449 outside. The benefit is that you can use more than one VLAN on a
456 To allow host communication with an isolated network. It is possible
457 to apply VLAN tags to any network device (NIC, Bond, Bridge). In
458 general, you should configure the VLAN on the interface with the least
459 abstraction layers between itself and the physical NIC.
461 For example, in a default configuration where you want to place
462 the host management address on a separate VLAN.
465 .Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge
468 iface lo inet loopback
470 iface eno1 inet manual
472 iface eno1.5 inet manual
475 iface vmbr0v5 inet static
477 netmask 255.255.255.0
484 iface vmbr0 inet manual
491 .Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
494 iface lo inet loopback
496 iface eno1 inet manual
500 iface vmbr0.5 inet static
502 netmask 255.255.255.0
506 iface vmbr0 inet manual
510 bridge-vlan-aware yes
513 The next example is the same setup but a bond is used to
514 make this network fail-safe.
516 .Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge
519 iface lo inet loopback
521 iface eno1 inet manual
523 iface eno2 inet manual
526 iface bond0 inet manual
527 bond-slaves eno1 eno2
530 bond-xmit-hash-policy layer2+3
532 iface bond0.5 inet manual
535 iface vmbr0v5 inet static
537 netmask 255.255.255.0
544 iface vmbr0 inet manual
551 Disabling IPv6 on the Node
552 ~~~~~~~~~~~~~~~~~~~~~~~~~~
554 {pve} works correctly in all environments, irrespective of whether IPv6 is
555 deployed or not. We recommend leaving all settings at the provided defaults.
557 Should you still need to disable support for IPv6 on your node, do so by
558 creating an appropriate `sysctl.conf (5)` snippet file and setting the proper
559 https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt[sysctls],
560 for example adding `/etc/sysctl.d/disable-ipv6.conf` with content:
563 net.ipv6.conf.all.disable_ipv6 = 1
564 net.ipv6.conf.default.disable_ipv6 = 1
567 This method is preferred to disabling the loading of the IPv6 module on the
568 https://www.kernel.org/doc/Documentation/networking/ipv6.rst[kernel commandline].
571 TODO: explain IPv6 support?