1 [[sysadmin_network_configuration]]
8 Network configuration can be done either via the GUI, or by manually
9 editing the file `/etc/network/interfaces`, which contains the
10 whole network configuration. The `interfaces(5)` manual page contains the
11 complete format description. All {pve} tools try hard to keep direct
12 user modifications, but using the GUI is still preferable, because it
13 protects you from errors.
15 Once the network is configured, you can use the Debian traditional tools `ifup`
16 and `ifdown` commands to bring interfaces up and down.
21 {pve} does not write changes directly to `/etc/network/interfaces`. Instead, we
22 write into a temporary file called `/etc/network/interfaces.new`, this way you
23 can do many related changes at once. This also allows to ensure your changes
24 are correct before applying, as a wrong network configuration may render a node
30 With the default installed `ifupdown` network managing package you need to
31 reboot to commit any pending network changes. Most of the time, the basic {pve}
32 network setup is stable and does not change often, so rebooting should not be
35 Reload Network with ifupdown2
36 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38 With the optional `ifupdown2` network managing package you also can reload the
39 network configuration live, without requiring a reboot.
41 NOTE: 'ifupdown2' cannot understand 'OpenVSwitch' syntax, so reloading is *not*
42 possible if OVS interfaces are configured.
44 Since {pve} 6.1 you can apply pending network changes over the web-interface,
45 using the 'Apply Configuration' button in the 'Network' panel of a node.
47 To install 'ifupdown2' ensure you have the latest {pve} updates installed, then
49 WARNING: installing 'ifupdown2' will remove 'ifupdown', but as the removal
50 scripts of 'ifupdown' before version '0.8.35+pve1' have a issue where network
51 is fully stopped on removal footnote:[Introduced with Debian Buster:
52 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945877] you *must* ensure
53 that you have a up to date 'ifupdown' package version.
55 For the installation itself you can then simply do:
59 With that you're all set. You can also switch back to the 'ifupdown' variant at
60 any time, if you run into issues.
65 We currently use the following naming conventions for device names:
67 * Ethernet devices: en*, systemd network interface names. This naming scheme is
68 used for new {pve} installations since version 5.0.
70 * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) This naming
71 scheme is used for {pve} hosts which were installed before the 5.0
72 release. When upgrading to 5.0, the names are kept as-is.
74 * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`)
76 * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...)
78 * VLANs: Simply add the VLAN number to the device name,
79 separated by a period (`eno1.50`, `bond1.30`)
81 This makes it easier to debug networks problems, because the device
82 name implies the device type.
84 Systemd Network Interface Names
85 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
87 Systemd uses the two character prefix 'en' for Ethernet network
88 devices. The next characters depends on the device driver and the fact
89 which schema matches first.
91 * o<index>[n<phys_port_name>|d<dev_port>] — devices on board
93 * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id
95 * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id
97 * x<MAC> — device by MAC address
99 The most common patterns are:
101 * eno1 — is the first on board NIC
103 * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.
105 For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names].
107 Choosing a network configuration
108 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
110 Depending on your current network organization and your resources you can
111 choose either a bridged, routed, or masquerading networking setup.
113 {pve} server in a private LAN, using an external gateway to reach the internet
114 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
116 The *Bridged* model makes the most sense in this case, and this is also
117 the default mode on new {pve} installations.
118 Each of your Guest system will have a virtual interface attached to the
119 {pve} bridge. This is similar in effect to having the Guest network card
120 directly connected to a new switch on your LAN, the {pve} host playing the role
123 {pve} server at hosting provider, with public IP ranges for Guests
124 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
126 For this setup, you can use either a *Bridged* or *Routed* model, depending on
127 what your provider allows.
129 {pve} server at hosting provider, with a single public IP address
130 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
132 In that case the only way to get outgoing network accesses for your guest
133 systems is to use *Masquerading*. For incoming network access to your guests,
134 you will need to configure *Port Forwarding*.
136 For further flexibility, you can configure
137 VLANs (IEEE 802.1q) and network bonding, also known as "link
138 aggregation". That way it is possible to build complex and flexible
141 Default Configuration using a Bridge
142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
144 [thumbnail="default-network-setup-bridge.svg"]
145 Bridges are like physical network switches implemented in software.
146 All virtual guests can share a single bridge, or you can create multiple
147 bridges to separate network domains. Each host can have up to 4094 bridges.
149 The installation program creates a single bridge named `vmbr0`, which
150 is connected to the first Ethernet card. The corresponding
151 configuration in `/etc/network/interfaces` might look like this:
155 iface lo inet loopback
157 iface eno1 inet manual
160 iface vmbr0 inet static
162 netmask 255.255.255.0
169 Virtual machines behave as if they were directly connected to the
170 physical network. The network, in turn, sees each virtual machine as
171 having its own MAC, even though there is only one network cable
172 connecting all of these VMs to the network.
177 Most hosting providers do not support the above setup. For security
178 reasons, they disable networking as soon as they detect multiple MAC
179 addresses on a single interface.
181 TIP: Some providers allows you to register additional MACs on their
182 management interface. This avoids the problem, but is clumsy to
183 configure because you need to register a MAC for each of your VMs.
185 You can avoid the problem by ``routing'' all traffic via a single
186 interface. This makes sure that all network packets use the same MAC
189 [thumbnail="default-network-setup-routed.svg"]
190 A common scenario is that you have a public IP (assume `198.51.100.5`
191 for this example), and an additional IP block for your VMs
192 (`203.0.113.16/29`). We recommend the following setup for such
197 iface lo inet loopback
200 iface eno1 inet static
202 netmask 255.255.255.0
204 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
205 post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
209 iface vmbr0 inet static
211 netmask 255.255.255.248
218 Masquerading (NAT) with `iptables`
219 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
221 Masquerading allows guests having only a private IP address to access the
222 network by using the host IP address for outgoing traffic. Each outgoing
223 packet is rewritten by `iptables` to appear as originating from the host,
224 and responses are rewritten accordingly to be routed to the original sender.
228 iface lo inet loopback
232 iface eno1 inet static
234 netmask 255.255.255.0
239 iface vmbr0 inet static
241 netmask 255.255.255.0
246 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
247 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
248 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
255 Bonding (also called NIC teaming or Link Aggregation) is a technique
256 for binding multiple NIC's to a single network device. It is possible
257 to achieve different goals, like make the network fault-tolerant,
258 increase the performance or both together.
260 High-speed hardware like Fibre Channel and the associated switching
261 hardware can be quite expensive. By doing link aggregation, two NICs
262 can appear as one logical interface, resulting in double speed. This
263 is a native Linux kernel feature that is supported by most
264 switches. If your nodes have multiple Ethernet ports, you can
265 distribute your points of failure by running network cables to
266 different switches and the bonded connection will failover to one
267 cable or the other in case of network trouble.
269 Aggregated links can improve live-migration delays and improve the
270 speed of replication of data between Proxmox VE Cluster nodes.
272 There are 7 modes for bonding:
274 * *Round-robin (balance-rr):* Transmit network packets in sequential
275 order from the first available network interface (NIC) slave through
276 the last. This mode provides load balancing and fault tolerance.
278 * *Active-backup (active-backup):* Only one NIC slave in the bond is
279 active. A different slave becomes active if, and only if, the active
280 slave fails. The single logical bonded interface's MAC address is
281 externally visible on only one NIC (port) to avoid distortion in the
282 network switch. This mode provides fault tolerance.
284 * *XOR (balance-xor):* Transmit network packets based on [(source MAC
285 address XOR'd with destination MAC address) modulo NIC slave
286 count]. This selects the same NIC slave for each destination MAC
287 address. This mode provides load balancing and fault tolerance.
289 * *Broadcast (broadcast):* Transmit network packets on all slave
290 network interfaces. This mode provides fault tolerance.
292 * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
293 aggregation groups that share the same speed and duplex
294 settings. Utilizes all slave network interfaces in the active
295 aggregator group according to the 802.3ad specification.
297 * *Adaptive transmit load balancing (balance-tlb):* Linux bonding
298 driver mode that does not require any special network-switch
299 support. The outgoing network packet traffic is distributed according
300 to the current load (computed relative to the speed) on each network
301 interface slave. Incoming traffic is received by one currently
302 designated slave network interface. If this receiving slave fails,
303 another slave takes over the MAC address of the failed receiving
306 * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive
307 load balancing (rlb) for IPV4 traffic, and does not require any
308 special network switch support. The receive load balancing is achieved
309 by ARP negotiation. The bonding driver intercepts the ARP Replies sent
310 by the local system on their way out and overwrites the source
311 hardware address with the unique hardware address of one of the NIC
312 slaves in the single logical bonded interface such that different
313 network-peers use different MAC addresses for their network packet
316 If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
317 the corresponding bonding mode (802.3ad). Otherwise you should generally use the
318 active-backup mode. +
319 // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html
320 If you intend to run your cluster network on the bonding interfaces, then you
321 have to use active-passive mode on the bonding interfaces, other modes are
324 The following bond configuration can be used as distributed/shared
325 storage network. The benefit would be that you get more speed and the
326 network will be fault-tolerant.
328 .Example: Use bond with fixed IP address
331 iface lo inet loopback
333 iface eno1 inet manual
335 iface eno2 inet manual
338 iface bond0 inet static
341 netmask 255.255.255.0
344 bond_xmit_hash_policy layer2+3
347 iface vmbr0 inet static
349 netmask 255.255.255.0
358 [thumbnail="default-network-setup-bond.svg"]
359 Another possibility it to use the bond directly as bridge port.
360 This can be used to make the guest network fault-tolerant.
362 .Example: Use a bond as bridge port
365 iface lo inet loopback
367 iface eno1 inet manual
369 iface eno2 inet manual
372 iface bond0 inet manual
376 bond_xmit_hash_policy layer2+3
379 iface vmbr0 inet static
381 netmask 255.255.255.0
393 A virtual LAN (VLAN) is a broadcast domain that is partitioned and
394 isolated in the network at layer two. So it is possible to have
395 multiple networks (4096) in a physical network, each independent of
398 Each VLAN network is identified by a number often called 'tag'.
399 Network packages are then 'tagged' to identify which virtual network
403 VLAN for Guest Networks
404 ^^^^^^^^^^^^^^^^^^^^^^^
406 {pve} supports this setup out of the box. You can specify the VLAN tag
407 when you create a VM. The VLAN tag is part of the guest network
408 configuration. The networking layer supports different modes to
409 implement VLANs, depending on the bridge configuration:
411 * *VLAN awareness on the Linux bridge:*
412 In this case, each guest's virtual network card is assigned to a VLAN tag,
413 which is transparently supported by the Linux bridge.
414 Trunk mode is also possible, but that makes configuration
415 in the guest necessary.
417 * *"traditional" VLAN on the Linux bridge:*
418 In contrast to the VLAN awareness method, this method is not transparent
419 and creates a VLAN device with associated bridge for each VLAN.
420 That is, creating a guest on VLAN 5 for example, would create two
421 interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs.
423 * *Open vSwitch VLAN:*
424 This mode uses the OVS VLAN feature.
426 * *Guest configured VLAN:*
427 VLANs are assigned inside the guest. In this case, the setup is
428 completely done inside the guest and can not be influenced from the
429 outside. The benefit is that you can use more than one VLAN on a
436 To allow host communication with an isolated network. It is possible
437 to apply VLAN tags to any network device (NIC, Bond, Bridge). In
438 general, you should configure the VLAN on the interface with the least
439 abstraction layers between itself and the physical NIC.
441 For example, in a default configuration where you want to place
442 the host management address on a separate VLAN.
445 .Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge
448 iface lo inet loopback
450 iface eno1 inet manual
452 iface eno1.5 inet manual
455 iface vmbr0v5 inet static
457 netmask 255.255.255.0
464 iface vmbr0 inet manual
471 .Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
474 iface lo inet loopback
476 iface eno1 inet manual
480 iface vmbr0.5 inet static
482 netmask 255.255.255.0
486 iface vmbr0 inet manual
490 bridge_vlan_aware yes
493 The next example is the same setup but a bond is used to
494 make this network fail-safe.
496 .Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge
499 iface lo inet loopback
501 iface eno1 inet manual
503 iface eno2 inet manual
506 iface bond0 inet manual
510 bond_xmit_hash_policy layer2+3
512 iface bond0.5 inet manual
515 iface vmbr0v5 inet static
517 netmask 255.255.255.0
524 iface vmbr0 inet manual
532 TODO: explain IPv6 support?