:pve-toplevel:
endif::wiki[]
-Network configuration can be done either via the GUI, or by manually
-editing the file `/etc/network/interfaces`, which contains the
-whole network configuration. The `interfaces(5)` manual page contains the
-complete format description. All {pve} tools try hard to keep direct
- user modifications, but using the GUI is still preferable, because it
+{pve} is using the Linux network stack. This provides a lot of flexibility on
+how to set up the network on the {pve} nodes. The configuration can be done
+either via the GUI, or by manually editing the file `/etc/network/interfaces`,
+which contains the whole network configuration. The `interfaces(5)` manual
+page contains the complete format description. All {pve} tools try hard to keep
+direct user modifications, but using the GUI is still preferable, because it
protects you from errors.
-Once the network is configured, you can use the Debian traditional tools `ifup`
-and `ifdown` commands to bring interfaces up and down.
+A 'vmbr' interface is needed to connect guests to the underlying physical
+network. They are a Linux bridge which can be thought of as a virtual switch
+to which the guests and physical interfaces are connected to. This section
+provides some examples on how the network can be set up to accomodate different
+use cases like redundancy with a xref:sysadmin_network_bond['bond'],
+xref:sysadmin_network_vlan['vlans'] or
+xref:sysadmin_network_routed['routed'] and
+xref:sysadmin_network_masquerading['NAT'] setups.
-NOTE: {pve} does not write changes directly to
-`/etc/network/interfaces`. Instead, we write into a temporary file
-called `/etc/network/interfaces.new`, and commit those changes when
-you reboot the node.
+The xref:chapter_pvesdn[Software Defined Network] is an option for more complex
+virtual networks in {pve} clusters.
+
+WARNING: It's discouraged to use the traditional Debian tools `ifup` and `ifdown`
+if unsure, as they have some pitfalls like interupting all guest traffic on
+`ifdown vmbrX` but not reconnecting those guest again when doing `ifup` on the
+same bridge later.
+
+Apply Network Changes
+~~~~~~~~~~~~~~~~~~~~~
+
+{pve} does not write changes directly to `/etc/network/interfaces`. Instead, we
+write into a temporary file called `/etc/network/interfaces.new`, this way you
+can do many related changes at once. This also allows to ensure your changes
+are correct before applying, as a wrong network configuration may render a node
+inaccessible.
+
+Live-Reload Network with ifupdown2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+With the recommended 'ifupdown2' package (default for new installations since
+{pve} 7.0), it is possible to apply network configuration changes without a
+reboot. If you change the network configuration via the GUI, you can click the
+'Apply Configuration' button. This will move changes from the staging
+`interfaces.new` file to `/etc/network/interfaces` and apply them live.
+
+If you made manual changes directly to the `/etc/network/interfaces` file, you
+can apply them by running `ifreload -a`
+
+NOTE: If you installed {pve} on top of Debian, or upgraded to {pve} 7.0 from an
+older {pve} installation, make sure 'ifupdown2' is installed: `apt install
+ifupdown2`
+
+Reboot Node to Apply
+^^^^^^^^^^^^^^^^^^^^
+
+Another way to apply a new network configuration is to reboot the node.
+In that case the systemd service `pvenetcommit` will activate the staging
+`interfaces.new` file before the `networking` service will apply that
+configuration.
Naming Conventions
~~~~~~~~~~~~~~~~~~
Choosing a network configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Depending on your current network organization and your resources you can
+Depending on your current network organization and your resources you can
choose either a bridged, routed, or masquerading networking setup.
{pve} server in a private LAN, using an external gateway to reach the internet
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The *Bridged* model makes the most sense in this case, and this is also
+The *Bridged* model makes the most sense in this case, and this is also
the default mode on new {pve} installations.
-Each of your Guest system will have a virtual interface attached to the
-{pve} bridge. This is similar in effect to having the Guest network card
+Each of your Guest system will have a virtual interface attached to the
+{pve} bridge. This is similar in effect to having the Guest network card
directly connected to a new switch on your LAN, the {pve} host playing the role
of the switch.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In that case the only way to get outgoing network accesses for your guest
-systems is to use *Masquerading*. For incoming network access to your guests,
+systems is to use *Masquerading*. For incoming network access to your guests,
you will need to configure *Port Forwarding*.
For further flexibility, you can configure
Default Configuration using a Bridge
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+[thumbnail="default-network-setup-bridge.svg"]
Bridges are like physical network switches implemented in software.
-All VMs can share a single bridge, or you can create multiple bridges to
-separate network domains. Each host can have up to 4094 bridges.
+All virtual guests can share a single bridge, or you can create multiple
+bridges to separate network domains. Each host can have up to 4094 bridges.
The installation program creates a single bridge named `vmbr0`, which
is connected to the first Ethernet card. The corresponding
auto vmbr0
iface vmbr0 inet static
- address 192.168.10.2
- netmask 255.255.255.0
+ address 192.168.10.2/24
gateway 192.168.10.1
- bridge_ports eno1
- bridge_stp off
- bridge_fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
----
Virtual machines behave as if they were directly connected to the
having its own MAC, even though there is only one network cable
connecting all of these VMs to the network.
+[[sysadmin_network_routed]]
Routed Configuration
~~~~~~~~~~~~~~~~~~~~
reasons, they disable networking as soon as they detect multiple MAC
addresses on a single interface.
-TIP: Some providers allows you to register additional MACs on there
-management interface. This avoids the problem, but is clumsy to
+TIP: Some providers allow you to register additional MACs through their
+management interface. This avoids the problem, but can be clumsy to
configure because you need to register a MAC for each of your VMs.
You can avoid the problem by ``routing'' all traffic via a single
interface. This makes sure that all network packets use the same MAC
address.
+[thumbnail="default-network-setup-routed.svg"]
A common scenario is that you have a public IP (assume `198.51.100.5`
for this example), and an additional IP block for your VMs
-(`203.0.113.16/29`). We recommend the following setup for such
+(`203.0.113.16/28`). We recommend the following setup for such
situations:
----
auto lo
iface lo inet loopback
-auto eno1
-iface eno1 inet static
- address 198.51.100.5
- netmask 255.255.255.0
+auto eno0
+iface eno0 inet static
+ address 198.51.100.5/29
gateway 198.51.100.1
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
- post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
+ post-up echo 1 > /proc/sys/net/ipv4/conf/eno0/proxy_arp
auto vmbr0
iface vmbr0 inet static
- address 203.0.113.17
- netmask 255.255.255.248
- bridge_ports none
- bridge_stp off
- bridge_fd 0
+ address 203.0.113.17/28
+ bridge-ports none
+ bridge-stp off
+ bridge-fd 0
----
+[[sysadmin_network_masquerading]]
Masquerading (NAT) with `iptables`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
auto eno1
#real IP address
iface eno1 inet static
- address 198.51.100.5
- netmask 255.255.255.0
+ address 198.51.100.5/24
gateway 198.51.100.1
auto vmbr0
#private sub network
iface vmbr0 inet static
- address 10.10.10.1
- netmask 255.255.255.0
- bridge_ports none
- bridge_stp off
- bridge_fd 0
+ address 10.10.10.1/24
+ bridge-ports none
+ bridge-stp off
+ bridge-fd 0
- post-up echo 1 > /proc/sys/net/ipv4/ip_forward
+ post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
----
+NOTE: In some masquerade setups with firewall enabled, conntrack zones might be
+needed for outgoing connections. Otherwise the firewall could block outgoing
+connections since they will prefer the `POSTROUTING` of the VM bridge (and not
+`MASQUERADE`).
+
+Adding these lines in the `/etc/network/interfaces` can fix this problem:
+
+----
+post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
+post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
+----
+
+For more information about this, refer to the following links:
+
+https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg[Netfilter Packet Flow]
+
+https://lwn.net/Articles/370152/[Patch on netdev-list introducing conntrack zones]
+https://web.archive.org/web/20220610151210/https://blog.lobraun.de/2019/05/19/prox/[Blog post with a good explanation by using TRACE in the raw table]
+
+
+[[sysadmin_network_bond]]
Linux Bond
~~~~~~~~~~
traffic.
If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
-the corresponding bonding mode (802.3ad). Otherwise you should generally use the
+the corresponding bonding mode (802.3ad). Otherwise you should generally use the
active-backup mode. +
// http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html
If you intend to run your cluster network on the bonding interfaces, then you
iface eno2 inet manual
+iface eno3 inet manual
+
auto bond0
iface bond0 inet static
- slaves eno1 eno2
- address 192.168.1.2
- netmask 255.255.255.0
- bond_miimon 100
- bond_mode 802.3ad
- bond_xmit_hash_policy layer2+3
+ bond-slaves eno1 eno2
+ address 192.168.1.2/24
+ bond-miimon 100
+ bond-mode 802.3ad
+ bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet static
- address 10.10.10.2
- netmask 255.255.255.0
+ address 10.10.10.2/24
gateway 10.10.10.1
- bridge_ports eno1
- bridge_stp off
- bridge_fd 0
+ bridge-ports eno3
+ bridge-stp off
+ bridge-fd 0
----
+[thumbnail="default-network-setup-bond.svg"]
Another possibility it to use the bond directly as bridge port.
This can be used to make the guest network fault-tolerant.
auto bond0
iface bond0 inet manual
- slaves eno1 eno2
- bond_miimon 100
- bond_mode 802.3ad
- bond_xmit_hash_policy layer2+3
+ bond-slaves eno1 eno2
+ bond-miimon 100
+ bond-mode 802.3ad
+ bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet static
- address 10.10.10.2
- netmask 255.255.255.0
+ address 10.10.10.2/24
gateway 10.10.10.1
- bridge_ports bond0
- bridge_stp off
- bridge_fd 0
+ bridge-ports bond0
+ bridge-stp off
+ bridge-fd 0
----
+
+[[sysadmin_network_vlan]]
VLAN 802.1Q
~~~~~~~~~~~
-A virtual LAN (VLAN) is any broadcast domain that is partitioned
-and isolated in the network at layer 2.
-So it is possible to have multiple networks (4096) in a physical network,
-each independent of the other ones.
-Each VLAN network is identified by a number often called `tag`.
-Network packages are then `tagged` to identify which virtual
-network they belong to.
+A virtual LAN (VLAN) is a broadcast domain that is partitioned and
+isolated in the network at layer two. So it is possible to have
+multiple networks (4096) in a physical network, each independent of
+the other ones.
-One or more VLANs can be used at any network device (Nic, Bond, Bridge).
-VLANs can be configured in several ways. Here, only the most common ones get
-described. We assume a network infrastructure based on Linux Kernel Networking
-(opposed to, e.g., Open vSwitch).
-Of course, there are scenarios that are not possible with this configuration,
-but it will work for most standard setups.
+Each VLAN network is identified by a number often called 'tag'.
+Network packages are then 'tagged' to identify which virtual network
+they belong to.
-Two of the most common and popular usage scenarios are:
-1.) VLAN for the guest networks.
-Proxmox supports three different ways of using VLAN in guests:
+VLAN for Guest Networks
+^^^^^^^^^^^^^^^^^^^^^^^
-* *VLAN awareness on the Linux Bridge:*
+{pve} supports this setup out of the box. You can specify the VLAN tag
+when you create a VM. The VLAN tag is part of the guest network
+configuration. The networking layer supports different modes to
+implement VLANs, depending on the bridge configuration:
+
+* *VLAN awareness on the Linux bridge:*
In this case, each guest's virtual network card is assigned to a VLAN tag,
-which is transparently supported by the Linux Bridge.
-Trunk mode is also possible, but that makes the configuration
+which is transparently supported by the Linux bridge.
+Trunk mode is also possible, but that makes configuration
in the guest necessary.
* *"traditional" VLAN on the Linux bridge:*
In contrast to the VLAN awareness method, this method is not transparent
and creates a VLAN device with associated bridge for each VLAN.
-That is, if e.g. in our default network, a guest VLAN 5 is used
-to create eno1.5 and vmbr0v5, which remains until rebooting.
+That is, creating a guest on VLAN 5 for example, would create two
+interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs.
+
+* *Open vSwitch VLAN:*
+This mode uses the OVS VLAN feature.
+
+* *Guest configured VLAN:*
+VLANs are assigned inside the guest. In this case, the setup is
+completely done inside the guest and can not be influenced from the
+outside. The benefit is that you can use more than one VLAN on a
+single virtual NIC.
+
-* *Guest configured:* The VLANs are assigned in the guest.
-In this case, the setup is in the guest and can not be influenced from the
-outside.
-The benefit is more then one VLAN on a single virtual NIC can be used.
+VLAN on the Host
+^^^^^^^^^^^^^^^^
-2.) VLAN on the host, to allow the host communication whit an isolated network.
-As already mentioned, it is possible to apply the VLAN to all network devices.
-In general, you should configure the VLAN on the interface with the least
+To allow host communication with an isolated network. It is possible
+to apply VLAN tags to any network device (NIC, Bond, Bridge). In
+general, you should configure the VLAN on the interface with the least
abstraction layers between itself and the physical NIC.
For example, in a default configuration where you want to place
the host management address on a separate VLAN.
-NOTE: In the examples we use the VLAN at bridge level to ensure the correct
-function of VLAN 5 in the guest network, but in combination with VLAN anwareness
-bridge this it will not work for guest network VLAN 5.
-The downside of this setup is more CPU usage.
-.Example: Use VLAN 5 for the {pve} management IP
+.Example: Use VLAN 5 for the {pve} management IP with traditional Linux bridge
----
auto lo
iface lo inet loopback
auto vmbr0v5
iface vmbr0v5 inet static
- address 10.10.10.2
- netmask 255.255.255.0
+ address 10.10.10.2/24
gateway 10.10.10.1
- bridge_ports eno1.5
- bridge_stp off
- bridge_fd 0
+ bridge-ports eno1.5
+ bridge-stp off
+ bridge-fd 0
auto vmbr0
iface vmbr0 inet manual
- bridge_ports eno1
- bridge_stp off
- bridge_fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
+
+----
+
+.Example: Use VLAN 5 for the {pve} management IP with VLAN aware Linux bridge
+----
+auto lo
+iface lo inet loopback
+
+iface eno1 inet manual
+
+
+auto vmbr0.5
+iface vmbr0.5 inet static
+ address 10.10.10.2/24
+ gateway 10.10.10.1
+auto vmbr0
+iface vmbr0 inet manual
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
+ bridge-vlan-aware yes
+ bridge-vids 2-4094
----
The next example is the same setup but a bond is used to
make this network fail-safe.
-.Example: Use VLAN 5 with bond0 for the {pve} management IP
+.Example: Use VLAN 5 with bond0 for the {pve} management IP with traditional Linux bridge
----
auto lo
iface lo inet loopback
auto bond0
iface bond0 inet manual
- slaves eno1 eno2
- bond_miimon 100
- bond_mode 802.3ad
- bond_xmit_hash_policy layer2+3
+ bond-slaves eno1 eno2
+ bond-miimon 100
+ bond-mode 802.3ad
+ bond-xmit-hash-policy layer2+3
iface bond0.5 inet manual
auto vmbr0v5
iface vmbr0v5 inet static
- address 10.10.10.2
- netmask 255.255.255.0
+ address 10.10.10.2/24
gateway 10.10.10.1
- bridge_ports bond0.5
- bridge_stp off
- bridge_fd 0
+ bridge-ports bond0.5
+ bridge-stp off
+ bridge-fd 0
auto vmbr0
iface vmbr0 inet manual
- bridge_ports bond0
- bridge_stp off
- bridge_fd 0
+ bridge-ports bond0
+ bridge-stp off
+ bridge-fd 0
+
+----
+
+Disabling IPv6 on the Node
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+{pve} works correctly in all environments, irrespective of whether IPv6 is
+deployed or not. We recommend leaving all settings at the provided defaults.
+
+Should you still need to disable support for IPv6 on your node, do so by
+creating an appropriate `sysctl.conf (5)` snippet file and setting the proper
+https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt[sysctls],
+for example adding `/etc/sysctl.d/disable-ipv6.conf` with content:
----
+net.ipv6.conf.all.disable_ipv6 = 1
+net.ipv6.conf.default.disable_ipv6 = 1
+----
+
+This method is preferred to disabling the loading of the IPv6 module on the
+https://www.kernel.org/doc/Documentation/networking/ipv6.rst[kernel commandline].
+
+
+Disabling MAC Learning on a Bridge
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, MAC learning is enabled on a bridge to ensure a smooth experience
+with virtual guests and their networks.
+
+But in some environments this can be undesired. Since {pve} 7.3 you can disable
+MAC learning on the bridge by setting the `bridge-disable-mac-learning 1`
+configuration on a bridge in `/etc/network/interfaces', for example:
+
+----
+# ...
+
+auto vmbr0
+iface vmbr0 inet static
+ address 10.10.10.2/24
+ gateway 10.10.10.1
+ bridge-ports ens18
+ bridge-stp off
+ bridge-fd 0
+ bridge-disable-mac-learning 1
+----
+
+Once enabled, {pve} will manually add the configured MAC address from VMs and
+Containers to the bridges forwarding database to ensure that guest can still
+use the network - but only when they are using their actual MAC address.
////
TODO: explain IPv6 support?
projects / pve-docs.git / blobdiff