pve-network.adoc

   1 [[sysadmin_network_configuration]]
   2 Network Configuration
   3 ---------------------
   4 include::attributes.txt[]
   5 ifdef::wiki[]
   6 :pve-toplevel:
   7 endif::wiki[]
   8
   9 {pve} uses a bridged networking model. Each host can have up to 4094
  10 bridges. Bridges are like physical network switches implemented in
  11 software. All VMs can share a single bridge, as if
  12 virtual network cables from each guest were all plugged into the same
  13 switch. But you can also create multiple bridges to separate network
  14 domains.
  15
  16 For connecting VMs to the outside world, bridges are attached to
  17 physical network cards. For further flexibility, you can configure
  18 VLANs (IEEE 802.1q) and network bonding, also known as "link
  19 aggregation". That way it is possible to build complex and flexible
  20 virtual networks.
  21
  22 Debian traditionally uses the `ifup` and `ifdown` commands to
  23 configure the network. The file `/etc/network/interfaces` contains the
  24 whole network setup. Please refer to to manual page (`man interfaces`)
  25 for a complete format description.
  26
  27 NOTE: {pve} does not write changes directly to
  28 `/etc/network/interfaces`. Instead, we write into a temporary file
  29 called `/etc/network/interfaces.new`, and commit those changes when
  30 you reboot the node.
  31
  32 It is worth mentioning that you can directly edit the configuration
  33 file. All {pve} tools tries hard to keep such direct user
  34 modifications. Using the GUI is still preferable, because it
  35 protect you from errors.
  36
  37
  38 Naming Conventions
  39 ~~~~~~~~~~~~~~~~~~
  40
  41 We currently use the following naming conventions for device names:
  42
  43 * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...)
  44
  45 * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`)
  46
  47 * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...)
  48
  49 * VLANs: Simply add the VLAN number to the device name,
  50   separated by a period (`eth0.50`, `bond1.30`)
  51
  52 This makes it easier to debug networks problems, because the device
  53 names implies the device type.
  54
  55 Default Configuration using a Bridge
  56 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  57
  58 The installation program creates a single bridge named `vmbr0`, which
  59 is connected to the first ethernet card `eth0`. The corresponding
  60 configuration in `/etc/network/interfaces` looks like this:
  61
  62 ----
  63 auto lo
  64 iface lo inet loopback
  65
  66 iface eth0 inet manual
  67
  68 auto vmbr0
  69 iface vmbr0 inet static
  70         address 192.168.10.2
  71         netmask 255.255.255.0
  72         gateway 192.168.10.1
  73         bridge_ports eth0
  74         bridge_stp off
  75         bridge_fd 0
  76 ----
  77
  78 Virtual machines behave as if they were directly connected to the
  79 physical network. The network, in turn, sees each virtual machine as
  80 having its own MAC, even though there is only one network cable
  81 connecting all of these VMs to the network.
  82
  83
  84 Routed Configuration
  85 ~~~~~~~~~~~~~~~~~~~~
  86
  87 Most hosting providers do not support the above setup. For security
  88 reasons, they disable networking as soon as they detect multiple MAC
  89 addresses on a single interface.
  90
  91 TIP: Some providers allows you to register additional MACs on there
  92 management interface. This avoids the problem, but is clumsy to
  93 configure because you need to register a MAC for each of your VMs.
  94
  95 You can avoid the problem by ``routing'' all traffic via a single
  96 interface. This makes sure that all network packets use the same MAC
  97 address.
  98
  99 A common scenario is that you have a public IP (assume `192.168.10.2`
 100 for this example), and an additional IP block for your VMs
 101 (`10.10.10.1/255.255.255.0`). We recommend the following setup for such
 102 situations:
 103
 104 ----
 105 auto lo
 106 iface lo inet loopback
 107
 108 auto eth0
 109 iface eth0 inet static
 110         address  192.168.10.2
 111         netmask  255.255.255.0
 112         gateway  192.168.10.1
 113         post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
 114
 115
 116 auto vmbr0
 117 iface vmbr0 inet static
 118         address  10.10.10.1
 119         netmask  255.255.255.0
 120         bridge_ports none
 121         bridge_stp off
 122         bridge_fd 0
 123 ----
 124
 125
 126 Masquerading (NAT) with `iptables`
 127 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 128
 129 In some cases you may want to use private IPs behind your Proxmox
 130 host's true IP, and masquerade the traffic using NAT:
 131
 132 ----
 133 auto lo
 134 iface lo inet loopback
 135
 136 auto eth0
 137 #real IP adress
 138 iface eth0 inet static
 139         address  192.168.10.2
 140         netmask  255.255.255.0
 141         gateway  192.168.10.1
 142
 143 auto vmbr0
 144 #private sub network
 145 iface vmbr0 inet static
 146         address  10.10.10.1
 147         netmask  255.255.255.0
 148         bridge_ports none
 149         bridge_stp off
 150         bridge_fd 0
 151
 152         post-up echo 1 > /proc/sys/net/ipv4/ip_forward
 153         post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE
 154         post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE
 155 ----
 156
 157
 158 Linux Bond
 159 ~~~~~~~~~~
 160
 161 Bonding (also called NIC teaming or Link Aggregation) is a technique
 162 for binding multiple NIC's to a single network device.  It is possible
 163 to achieve different goals, like make the network fault-tolerant,
 164 increase the performance or both together.
 165
 166 High-speed hardware like Fibre Channel and the associated switching
 167 hardware can be quite expensive. By doing link aggregation, two NICs
 168 can appear as one logical interface, resulting in double speed. This
 169 is a native Linux kernel feature that is supported by most
 170 switches. If your nodes have multiple Ethernet ports, you can
 171 distribute your points of failure by running network cables to
 172 different switches and the bonded connection will failover to one
 173 cable or the other in case of network trouble.
 174
 175 Aggregated links can improve live-migration delays and improve the
 176 speed of replication of data between Proxmox VE Cluster nodes.
 177
 178 There are 7 modes for bonding:
 179
 180 * *Round-robin (balance-rr):* Transmit network packets in sequential
 181 order from the first available network interface (NIC) slave through
 182 the last. This mode provides load balancing and fault tolerance.
 183
 184 * *Active-backup (active-backup):* Only one NIC slave in the bond is
 185 active. A different slave becomes active if, and only if, the active
 186 slave fails. The single logical bonded interface's MAC address is
 187 externally visible on only one NIC (port) to avoid distortion in the
 188 network switch. This mode provides fault tolerance.
 189
 190 * *XOR (balance-xor):* Transmit network packets based on [(source MAC
 191 address XOR'd with destination MAC address) modulo NIC slave
 192 count]. This selects the same NIC slave for each destination MAC
 193 address. This mode provides load balancing and fault tolerance.
 194
 195 * *Broadcast (broadcast):* Transmit network packets on all slave
 196 network interfaces. This mode provides fault tolerance.
 197
 198 * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
 199 aggregation groups that share the same speed and duplex
 200 settings. Utilizes all slave network interfaces in the active
 201 aggregator group according to the 802.3ad specification.
 202
 203 * *Adaptive transmit load balancing (balance-tlb):* Linux bonding
 204 driver mode that does not require any special network-switch
 205 support. The outgoing network packet traffic is distributed according
 206 to the current load (computed relative to the speed) on each network
 207 interface slave. Incoming traffic is received by one currently
 208 designated slave network interface. If this receiving slave fails,
 209 another slave takes over the MAC address of the failed receiving
 210 slave.
 211
 212 * *Adaptive load balancing (balanceIEEE 802.3ad Dynamic link
 213 aggregation (802.3ad)(LACP):-alb):* Includes balance-tlb plus receive
 214 load balancing (rlb) for IPV4 traffic, and does not require any
 215 special network switch support. The receive load balancing is achieved
 216 by ARP negotiation. The bonding driver intercepts the ARP Replies sent
 217 by the local system on their way out and overwrites the source
 218 hardware address with the unique hardware address of one of the NIC
 219 slaves in the single logical bonded interface such that different
 220 network-peers use different MAC addresses for their network packet
 221 traffic.
 222
 223 For the most setups the active-backup are the best choice or if your
 224 switch support LACP "IEEE 802.3ad" this mode should be preferred.
 225
 226 The following bond configuration can be used as distributed/shared
 227 storage network. The benefit would be that you get more speed and the
 228 network will be fault-tolerant.
 229
 230 .Example: Use bond with fixed IP address
 231 ----
 232 auto lo
 233 iface lo inet loopback
 234
 235 iface eth1 inet manual
 236
 237 iface eth2 inet manual
 238
 239 auto bond0
 240 iface bond0 inet static
 241       slaves eth1 eth2
 242       address  192.168.1.2
 243       netmask  255.255.255.0
 244       bond_miimon 100
 245       bond_mode 802.3ad
 246       bond_xmit_hash_policy layer2+3
 247
 248 auto vmbr0
 249 iface vmbr0 inet static
 250         address  10.10.10.2
 251         netmask  255.255.255.0
 252         gateway  10.10.10.1
 253         bridge_ports eth0
 254         bridge_stp off
 255         bridge_fd 0
 256
 257 ----
 258
 259
 260 Another possibility it to use the bond directly as bridge port.
 261 This can be used to make the guest network fault-tolerant.
 262
 263 .Example: Use a bond as bridge port
 264 ----
 265 auto lo
 266 iface lo inet loopback
 267
 268 iface eth1 inet manual
 269
 270 iface eth2 inet manual
 271
 272 auto bond0
 273 iface bond0 inet maunal
 274       slaves eth1 eth2
 275       bond_miimon 100
 276       bond_mode 802.3ad
 277       bond_xmit_hash_policy layer2+3
 278
 279 auto vmbr0
 280 iface vmbr0 inet static
 281         address  10.10.10.2
 282         netmask  255.255.255.0
 283         gateway  10.10.10.1
 284         bridge_ports bond0
 285         bridge_stp off
 286         bridge_fd 0
 287
 288 ----
 289
 290 ////
 291 TODO: explain IPv6 support?
 292 TODO: explan OVS
 293 ////