pve-network.adoc

   1 [[sysadmin_network_configuration]]
   2 Network Configuration
   3 ---------------------
   4 ifdef::wiki[]
   5 :pve-toplevel:
   6 endif::wiki[]
   7
   8 {pve} uses a bridged networking model. Each host can have up to 4094
   9 bridges. Bridges are like physical network switches implemented in
  10 software. All VMs can share a single bridge, as if
  11 virtual network cables from each guest were all plugged into the same
  12 switch. But you can also create multiple bridges to separate network
  13 domains.
  14
  15 For connecting VMs to the outside world, bridges are attached to
  16 physical network cards. For further flexibility, you can configure
  17 VLANs (IEEE 802.1q) and network bonding, also known as "link
  18 aggregation". That way it is possible to build complex and flexible
  19 virtual networks.
  20
  21 Debian traditionally uses the `ifup` and `ifdown` commands to
  22 configure the network. The file `/etc/network/interfaces` contains the
  23 whole network setup. Please refer to to manual page (`man interfaces`)
  24 for a complete format description.
  25
  26 NOTE: {pve} does not write changes directly to
  27 `/etc/network/interfaces`. Instead, we write into a temporary file
  28 called `/etc/network/interfaces.new`, and commit those changes when
  29 you reboot the node.
  30
  31 It is worth mentioning that you can directly edit the configuration
  32 file. All {pve} tools tries hard to keep such direct user
  33 modifications. Using the GUI is still preferable, because it
  34 protect you from errors.
  35
  36
  37 Naming Conventions
  38 ~~~~~~~~~~~~~~~~~~
  39
  40 We currently use the following naming conventions for device names:
  41
  42 * New Ethernet devices: en*, systemd network interface names.
  43
  44 * Legacy Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...)
  45 They are available when Proxmox VE has been updated by an earlier version.
  46
  47 * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`)
  48
  49 * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...)
  50
  51 * VLANs: Simply add the VLAN number to the device name,
  52   separated by a period (`eno1.50`, `bond1.30`)
  53
  54 This makes it easier to debug networks problems, because the device
  55 names implies the device type.
  56
  57
  58 Systemd Network Interface Names
  59 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  60
  61 Systemd uses the two character prefix 'en' for Ethernet network
  62 devices. The next characters depends on the device driver and the fact
  63 which schema matches first.
  64
  65 * o<index>[n<phys_port_name>|d<dev_port>] — devices on board
  66
  67 * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id
  68
  69 * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id
  70
  71 * x<MAC> — device by MAC address
  72
  73 The most common patterns are:
  74
  75 * eno1 — is the first on board NIC
  76
  77 * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.
  78
  79 For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names].
  80
  81
  82 Default Configuration using a Bridge
  83 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  84
  85 The installation program creates a single bridge named `vmbr0`, which
  86 is connected to the first Ethernet card. The corresponding
  87 configuration in `/etc/network/interfaces` might look like this:
  88
  89 ----
  90 auto lo
  91 iface lo inet loopback
  92
  93 iface eno1 inet manual
  94
  95 auto vmbr0
  96 iface vmbr0 inet static
  97         address 192.168.10.2
  98         netmask 255.255.255.0
  99         gateway 192.168.10.1
 100         bridge_ports eno1
 101         bridge_stp off
 102         bridge_fd 0
 103 ----
 104
 105 Virtual machines behave as if they were directly connected to the
 106 physical network. The network, in turn, sees each virtual machine as
 107 having its own MAC, even though there is only one network cable
 108 connecting all of these VMs to the network.
 109
 110
 111 Routed Configuration
 112 ~~~~~~~~~~~~~~~~~~~~
 113
 114 Most hosting providers do not support the above setup. For security
 115 reasons, they disable networking as soon as they detect multiple MAC
 116 addresses on a single interface.
 117
 118 TIP: Some providers allows you to register additional MACs on there
 119 management interface. This avoids the problem, but is clumsy to
 120 configure because you need to register a MAC for each of your VMs.
 121
 122 You can avoid the problem by ``routing'' all traffic via a single
 123 interface. This makes sure that all network packets use the same MAC
 124 address.
 125
 126 A common scenario is that you have a public IP (assume `192.168.10.2`
 127 for this example), and an additional IP block for your VMs
 128 (`10.10.10.1/255.255.255.0`). We recommend the following setup for such
 129 situations:
 130
 131 ----
 132 auto lo
 133 iface lo inet loopback
 134
 135 auto eno1
 136 iface eno1 inet static
 137         address  192.168.10.2
 138         netmask  255.255.255.0
 139         gateway  192.168.10.1
 140         post-up echo 1 > /proc/sys/net/ipv4/ip_forward
 141         post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
 142
 143
 144 auto vmbr0
 145 iface vmbr0 inet static
 146         address  10.10.10.1
 147         netmask  255.255.255.0
 148         bridge_ports none
 149         bridge_stp off
 150         bridge_fd 0
 151 ----
 152
 153
 154 Masquerading (NAT) with `iptables`
 155 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 156
 157 In some cases you may want to use private IPs behind your Proxmox
 158 host's true IP, and masquerade the traffic using NAT:
 159
 160 ----
 161 auto lo
 162 iface lo inet loopback
 163
 164 auto eno0
 165 #real IP address
 166 iface eno1 inet static
 167         address  192.168.10.2
 168         netmask  255.255.255.0
 169         gateway  192.168.10.1
 170
 171 auto vmbr0
 172 #private sub network
 173 iface vmbr0 inet static
 174         address  10.10.10.1
 175         netmask  255.255.255.0
 176         bridge_ports none
 177         bridge_stp off
 178         bridge_fd 0
 179
 180         post-up echo 1 > /proc/sys/net/ipv4/ip_forward
 181         post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
 182         post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
 183 ----
 184
 185
 186 Linux Bond
 187 ~~~~~~~~~~
 188
 189 Bonding (also called NIC teaming or Link Aggregation) is a technique
 190 for binding multiple NIC's to a single network device.  It is possible
 191 to achieve different goals, like make the network fault-tolerant,
 192 increase the performance or both together.
 193
 194 High-speed hardware like Fibre Channel and the associated switching
 195 hardware can be quite expensive. By doing link aggregation, two NICs
 196 can appear as one logical interface, resulting in double speed. This
 197 is a native Linux kernel feature that is supported by most
 198 switches. If your nodes have multiple Ethernet ports, you can
 199 distribute your points of failure by running network cables to
 200 different switches and the bonded connection will failover to one
 201 cable or the other in case of network trouble.
 202
 203 Aggregated links can improve live-migration delays and improve the
 204 speed of replication of data between Proxmox VE Cluster nodes.
 205
 206 There are 7 modes for bonding:
 207
 208 * *Round-robin (balance-rr):* Transmit network packets in sequential
 209 order from the first available network interface (NIC) slave through
 210 the last. This mode provides load balancing and fault tolerance.
 211
 212 * *Active-backup (active-backup):* Only one NIC slave in the bond is
 213 active. A different slave becomes active if, and only if, the active
 214 slave fails. The single logical bonded interface's MAC address is
 215 externally visible on only one NIC (port) to avoid distortion in the
 216 network switch. This mode provides fault tolerance.
 217
 218 * *XOR (balance-xor):* Transmit network packets based on [(source MAC
 219 address XOR'd with destination MAC address) modulo NIC slave
 220 count]. This selects the same NIC slave for each destination MAC
 221 address. This mode provides load balancing and fault tolerance.
 222
 223 * *Broadcast (broadcast):* Transmit network packets on all slave
 224 network interfaces. This mode provides fault tolerance.
 225
 226 * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
 227 aggregation groups that share the same speed and duplex
 228 settings. Utilizes all slave network interfaces in the active
 229 aggregator group according to the 802.3ad specification.
 230
 231 * *Adaptive transmit load balancing (balance-tlb):* Linux bonding
 232 driver mode that does not require any special network-switch
 233 support. The outgoing network packet traffic is distributed according
 234 to the current load (computed relative to the speed) on each network
 235 interface slave. Incoming traffic is received by one currently
 236 designated slave network interface. If this receiving slave fails,
 237 another slave takes over the MAC address of the failed receiving
 238 slave.
 239
 240 * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive
 241 load balancing (rlb) for IPV4 traffic, and does not require any
 242 special network switch support. The receive load balancing is achieved
 243 by ARP negotiation. The bonding driver intercepts the ARP Replies sent
 244 by the local system on their way out and overwrites the source
 245 hardware address with the unique hardware address of one of the NIC
 246 slaves in the single logical bonded interface such that different
 247 network-peers use different MAC addresses for their network packet
 248 traffic.
 249
 250 If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
 251 the corresponding bonding mode (802.3ad). Otherwise you should generally use the
 252 active-backup mode. +
 253 // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html
 254 If you intend to run your cluster network on the bonding interfaces, then you
 255 have to use active-passive mode on the bonding interfaces, other modes are
 256 unsupported.
 257
 258 The following bond configuration can be used as distributed/shared
 259 storage network. The benefit would be that you get more speed and the
 260 network will be fault-tolerant.
 261
 262 .Example: Use bond with fixed IP address
 263 ----
 264 auto lo
 265 iface lo inet loopback
 266
 267 iface eno1 inet manual
 268
 269 iface eno2 inet manual
 270
 271 auto bond0
 272 iface bond0 inet static
 273       slaves eno1 eno2
 274       address  192.168.1.2
 275       netmask  255.255.255.0
 276       bond_miimon 100
 277       bond_mode 802.3ad
 278       bond_xmit_hash_policy layer2+3
 279
 280 auto vmbr0
 281 iface vmbr0 inet static
 282         address  10.10.10.2
 283         netmask  255.255.255.0
 284         gateway  10.10.10.1
 285         bridge_ports eno1
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 ----
 290
 291
 292 Another possibility it to use the bond directly as bridge port.
 293 This can be used to make the guest network fault-tolerant.
 294
 295 .Example: Use a bond as bridge port
 296 ----
 297 auto lo
 298 iface lo inet loopback
 299
 300 iface eno1 inet manual
 301
 302 iface eno2 inet manual
 303
 304 auto bond0
 305 iface bond0 inet manual
 306       slaves eno1 eno2
 307       bond_miimon 100
 308       bond_mode 802.3ad
 309       bond_xmit_hash_policy layer2+3
 310
 311 auto vmbr0
 312 iface vmbr0 inet static
 313         address  10.10.10.2
 314         netmask  255.255.255.0
 315         gateway  10.10.10.1
 316         bridge_ports bond0
 317         bridge_stp off
 318         bridge_fd 0
 319
 320 ----
 321
 322 ////
 323 TODO: explain IPv6 support?
 324 TODO: explain OVS
 325 ////