1 [[sysadmin_network_configuration]]
8 Network configuration can be done either via the GUI, or by manually
9 editing the file `/etc/network/interfaces`, which contains the
10 whole network configuration. The `interfaces(5)` manual page contains the
11 complete format description. All {pve} tools try hard to keep direct
12 user modifications, but using the GUI is still preferable, because it
13 protects you from errors.
15 Once the network is configured, you can use the Debian traditional tools `ifup`
16 and `ifdown` commands to bring interfaces up and down.
18 NOTE: {pve} does not write changes directly to
19 `/etc/network/interfaces`. Instead, we write into a temporary file
20 called `/etc/network/interfaces.new`, and commit those changes when
26 We currently use the following naming conventions for device names:
28 * Ethernet devices: en*, systemd network interface names. This naming scheme is
29 used for new {pve} installations since version 5.0.
31 * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) This naming
32 scheme is used for {pve} hosts which were installed before the 5.0
33 release. When upgrading to 5.0, the names are kept as-is.
35 * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`)
37 * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...)
39 * VLANs: Simply add the VLAN number to the device name,
40 separated by a period (`eno1.50`, `bond1.30`)
42 This makes it easier to debug networks problems, because the device
43 name implies the device type.
45 Systemd Network Interface Names
46 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
48 Systemd uses the two character prefix 'en' for Ethernet network
49 devices. The next characters depends on the device driver and the fact
50 which schema matches first.
52 * o<index>[n<phys_port_name>|d<dev_port>] — devices on board
54 * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id
56 * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id
58 * x<MAC> — device by MAC address
60 The most common patterns are:
62 * eno1 — is the first on board NIC
64 * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.
66 For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names].
68 Choosing a network configuration
69 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71 Depending on your current network organization and your resources you can
72 choose either a bridged, routed, or masquerading networking setup.
74 {pve} server in a private LAN, using an external gateway to reach the internet
75 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
77 The *Bridged* model makes the most sense in this case, and this is also
78 the default mode on new {pve} installations.
79 Each of your Guest system will have a virtual interface attached to the
80 {pve} bridge. This is similar in effect to having the Guest network card
81 directly connected to a new switch on your LAN, the {pve} host playing the role
84 {pve} server at hosting provider, with public IP ranges for Guests
85 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
87 For this setup, you can use either a *Bridged* or *Routed* model, depending on
88 what your provider allows.
90 {pve} server at hosting provider, with a single public IP address
91 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
93 In that case the only way to get outgoing network accesses for your guest
94 systems is to use *Masquerading*. For incoming network access to your guests,
95 you will need to configure *Port Forwarding*.
97 For further flexibility, you can configure
98 VLANs (IEEE 802.1q) and network bonding, also known as "link
99 aggregation". That way it is possible to build complex and flexible
102 Default Configuration using a Bridge
103 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
105 [thumbnail="default-network-setup-bridge.svg"]
106 Bridges are like physical network switches implemented in software.
107 All VMs can share a single bridge, or you can create multiple bridges to
108 separate network domains. Each host can have up to 4094 bridges.
110 The installation program creates a single bridge named `vmbr0`, which
111 is connected to the first Ethernet card. The corresponding
112 configuration in `/etc/network/interfaces` might look like this:
116 iface lo inet loopback
118 iface eno1 inet manual
121 iface vmbr0 inet static
123 netmask 255.255.255.0
130 Virtual machines behave as if they were directly connected to the
131 physical network. The network, in turn, sees each virtual machine as
132 having its own MAC, even though there is only one network cable
133 connecting all of these VMs to the network.
138 Most hosting providers do not support the above setup. For security
139 reasons, they disable networking as soon as they detect multiple MAC
140 addresses on a single interface.
142 TIP: Some providers allows you to register additional MACs on there
143 management interface. This avoids the problem, but is clumsy to
144 configure because you need to register a MAC for each of your VMs.
146 You can avoid the problem by ``routing'' all traffic via a single
147 interface. This makes sure that all network packets use the same MAC
150 [thumbnail="default-network-setup-routed.svg"]
151 A common scenario is that you have a public IP (assume `198.51.100.5`
152 for this example), and an additional IP block for your VMs
153 (`203.0.113.16/29`). We recommend the following setup for such
158 iface lo inet loopback
161 iface eno1 inet static
163 netmask 255.255.255.0
165 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
166 post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
170 iface vmbr0 inet static
172 netmask 255.255.255.248
179 Masquerading (NAT) with `iptables`
180 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
182 Masquerading allows guests having only a private IP address to access the
183 network by using the host IP address for outgoing traffic. Each outgoing
184 packet is rewritten by `iptables` to appear as originating from the host,
185 and responses are rewritten accordingly to be routed to the original sender.
189 iface lo inet loopback
193 iface eno1 inet static
195 netmask 255.255.255.0
200 iface vmbr0 inet static
202 netmask 255.255.255.0
207 post-up echo 1 > /proc/sys/net/ipv4/ip_forward
208 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
209 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
216 Bonding (also called NIC teaming or Link Aggregation) is a technique
217 for binding multiple NIC's to a single network device. It is possible
218 to achieve different goals, like make the network fault-tolerant,
219 increase the performance or both together.
221 High-speed hardware like Fibre Channel and the associated switching
222 hardware can be quite expensive. By doing link aggregation, two NICs
223 can appear as one logical interface, resulting in double speed. This
224 is a native Linux kernel feature that is supported by most
225 switches. If your nodes have multiple Ethernet ports, you can
226 distribute your points of failure by running network cables to
227 different switches and the bonded connection will failover to one
228 cable or the other in case of network trouble.
230 Aggregated links can improve live-migration delays and improve the
231 speed of replication of data between Proxmox VE Cluster nodes.
233 There are 7 modes for bonding:
235 * *Round-robin (balance-rr):* Transmit network packets in sequential
236 order from the first available network interface (NIC) slave through
237 the last. This mode provides load balancing and fault tolerance.
239 * *Active-backup (active-backup):* Only one NIC slave in the bond is
240 active. A different slave becomes active if, and only if, the active
241 slave fails. The single logical bonded interface's MAC address is
242 externally visible on only one NIC (port) to avoid distortion in the
243 network switch. This mode provides fault tolerance.
245 * *XOR (balance-xor):* Transmit network packets based on [(source MAC
246 address XOR'd with destination MAC address) modulo NIC slave
247 count]. This selects the same NIC slave for each destination MAC
248 address. This mode provides load balancing and fault tolerance.
250 * *Broadcast (broadcast):* Transmit network packets on all slave
251 network interfaces. This mode provides fault tolerance.
253 * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
254 aggregation groups that share the same speed and duplex
255 settings. Utilizes all slave network interfaces in the active
256 aggregator group according to the 802.3ad specification.
258 * *Adaptive transmit load balancing (balance-tlb):* Linux bonding
259 driver mode that does not require any special network-switch
260 support. The outgoing network packet traffic is distributed according
261 to the current load (computed relative to the speed) on each network
262 interface slave. Incoming traffic is received by one currently
263 designated slave network interface. If this receiving slave fails,
264 another slave takes over the MAC address of the failed receiving
267 * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive
268 load balancing (rlb) for IPV4 traffic, and does not require any
269 special network switch support. The receive load balancing is achieved
270 by ARP negotiation. The bonding driver intercepts the ARP Replies sent
271 by the local system on their way out and overwrites the source
272 hardware address with the unique hardware address of one of the NIC
273 slaves in the single logical bonded interface such that different
274 network-peers use different MAC addresses for their network packet
277 If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
278 the corresponding bonding mode (802.3ad). Otherwise you should generally use the
279 active-backup mode. +
280 // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html
281 If you intend to run your cluster network on the bonding interfaces, then you
282 have to use active-passive mode on the bonding interfaces, other modes are
285 The following bond configuration can be used as distributed/shared
286 storage network. The benefit would be that you get more speed and the
287 network will be fault-tolerant.
289 .Example: Use bond with fixed IP address
292 iface lo inet loopback
294 iface eno1 inet manual
296 iface eno2 inet manual
299 iface bond0 inet static
302 netmask 255.255.255.0
305 bond_xmit_hash_policy layer2+3
308 iface vmbr0 inet static
310 netmask 255.255.255.0
319 [thumbnail="default-network-setup-bond.svg"]
320 Another possibility it to use the bond directly as bridge port.
321 This can be used to make the guest network fault-tolerant.
323 .Example: Use a bond as bridge port
326 iface lo inet loopback
328 iface eno1 inet manual
330 iface eno2 inet manual
333 iface bond0 inet manual
337 bond_xmit_hash_policy layer2+3
340 iface vmbr0 inet static
342 netmask 255.255.255.0
354 A virtual LAN (VLAN) is a broadcast domain that is partitioned and
355 isolated in the network at layer two. So it is possible to have
356 multiple networks (4096) in a physical network, each independent of
359 Each VLAN network is identified by a number often called 'tag'.
360 Network packages are then 'tagged' to identify which virtual network
364 VLAN for Guest Networks
365 ^^^^^^^^^^^^^^^^^^^^^^^
367 {pve} supports this setup out of the box. You can specify the VLAN tag
368 when you create a VM. The VLAN tag is part of the guest network
369 confinuration. The networking layer supports differnet modes to
370 implement VLANs, depending on the bridge configuration:
372 * *VLAN awareness on the Linux bridge:*
373 In this case, each guest's virtual network card is assigned to a VLAN tag,
374 which is transparently supported by the Linux bridge.
375 Trunk mode is also possible, but that makes the configuration
376 in the guest necessary.
378 * *"traditional" VLAN on the Linux bridge:*
379 In contrast to the VLAN awareness method, this method is not transparent
380 and creates a VLAN device with associated bridge for each VLAN.
381 That is, if e.g. in our default network, a guest VLAN 5 is used
382 to create eno1.5 and vmbr0v5, which remains until rebooting.
384 * *Open vSwitch VLAN:*
385 This mode uses the OVS VLAN feature.
387 * *Guest configured VLAN:*
388 VLANs are assigned inside the guest. In this case, the setup is
389 completely done inside the guest and can not be influenced from the
390 outside. The benefit is that you can use more than one VLAN on a
397 To allow host communication with an isolated network. It is possible
398 to apply VLAN tags to any network device (NIC, Bond, Bridge). In
399 general, you should configure the VLAN on the interface with the least
400 abstraction layers between itself and the physical NIC.
402 For example, in a default configuration where you want to place
403 the host management address on a separate VLAN.
406 .Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge
409 iface lo inet loopback
411 iface eno1 inet manual
413 iface eno1.5 inet manual
416 iface vmbr0v5 inet static
418 netmask 255.255.255.0
425 iface vmbr0 inet manual
432 .Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
435 iface lo inet loopback
437 iface eno1 inet manual
441 iface vmbr0.5 inet static
443 netmask 255.255.255.0
447 iface vmbr0 inet manual
451 bridge_vlan_aware yes
454 The next example is the same setup but a bond is used to
455 make this network fail-safe.
457 .Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge
460 iface lo inet loopback
462 iface eno1 inet manual
464 iface eno2 inet manual
467 iface bond0 inet manual
471 bond_xmit_hash_policy layer2+3
473 iface bond0.5 inet manual
476 iface vmbr0v5 inet static
478 netmask 255.255.255.0
485 iface vmbr0 inet manual
493 TODO: explain IPv6 support?