+ post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
+ post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
+----
+
+
+Linux Bond
+~~~~~~~~~~
+
+Bonding (also called NIC teaming or Link Aggregation) is a technique
+for binding multiple NIC's to a single network device. It is possible
+to achieve different goals, like make the network fault-tolerant,
+increase the performance or both together.
+
+High-speed hardware like Fibre Channel and the associated switching
+hardware can be quite expensive. By doing link aggregation, two NICs
+can appear as one logical interface, resulting in double speed. This
+is a native Linux kernel feature that is supported by most
+switches. If your nodes have multiple Ethernet ports, you can
+distribute your points of failure by running network cables to
+different switches and the bonded connection will failover to one
+cable or the other in case of network trouble.
+
+Aggregated links can improve live-migration delays and improve the
+speed of replication of data between Proxmox VE Cluster nodes.
+
+There are 7 modes for bonding:
+
+* *Round-robin (balance-rr):* Transmit network packets in sequential
+order from the first available network interface (NIC) slave through
+the last. This mode provides load balancing and fault tolerance.
+
+* *Active-backup (active-backup):* Only one NIC slave in the bond is
+active. A different slave becomes active if, and only if, the active
+slave fails. The single logical bonded interface's MAC address is
+externally visible on only one NIC (port) to avoid distortion in the
+network switch. This mode provides fault tolerance.
+
+* *XOR (balance-xor):* Transmit network packets based on [(source MAC
+address XOR'd with destination MAC address) modulo NIC slave
+count]. This selects the same NIC slave for each destination MAC
+address. This mode provides load balancing and fault tolerance.
+
+* *Broadcast (broadcast):* Transmit network packets on all slave
+network interfaces. This mode provides fault tolerance.
+
+* *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates
+aggregation groups that share the same speed and duplex
+settings. Utilizes all slave network interfaces in the active
+aggregator group according to the 802.3ad specification.
+
+* *Adaptive transmit load balancing (balance-tlb):* Linux bonding
+driver mode that does not require any special network-switch
+support. The outgoing network packet traffic is distributed according
+to the current load (computed relative to the speed) on each network
+interface slave. Incoming traffic is received by one currently
+designated slave network interface. If this receiving slave fails,
+another slave takes over the MAC address of the failed receiving
+slave.
+
+* *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive
+load balancing (rlb) for IPV4 traffic, and does not require any
+special network switch support. The receive load balancing is achieved
+by ARP negotiation. The bonding driver intercepts the ARP Replies sent
+by the local system on their way out and overwrites the source
+hardware address with the unique hardware address of one of the NIC
+slaves in the single logical bonded interface such that different
+network-peers use different MAC addresses for their network packet
+traffic.
+
+If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
+the corresponding bonding mode (802.3ad). Otherwise you should generally use the
+active-backup mode. +
+// http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html
+If you intend to run your cluster network on the bonding interfaces, then you
+have to use active-passive mode on the bonding interfaces, other modes are
+unsupported.
+
+The following bond configuration can be used as distributed/shared
+storage network. The benefit would be that you get more speed and the
+network will be fault-tolerant.
+
+.Example: Use bond with fixed IP address
+----
+auto lo
+iface lo inet loopback
+
+iface eno1 inet manual
+
+iface eno2 inet manual
+
+auto bond0
+iface bond0 inet static
+ slaves eno1 eno2
+ address 192.168.1.2
+ netmask 255.255.255.0
+ bond_miimon 100
+ bond_mode 802.3ad
+ bond_xmit_hash_policy layer2+3
+
+auto vmbr0
+iface vmbr0 inet static
+ address 10.10.10.2
+ netmask 255.255.255.0
+ gateway 10.10.10.1
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+----
+
+
+Another possibility it to use the bond directly as bridge port.
+This can be used to make the guest network fault-tolerant.
+
+.Example: Use a bond as bridge port
+----
+auto lo
+iface lo inet loopback
+
+iface eno1 inet manual
+
+iface eno2 inet manual
+
+auto bond0
+iface bond0 inet manual
+ slaves eno1 eno2
+ bond_miimon 100
+ bond_mode 802.3ad
+ bond_xmit_hash_policy layer2+3
+
+auto vmbr0
+iface vmbr0 inet static
+ address 10.10.10.2
+ netmask 255.255.255.0
+ gateway 10.10.10.1
+ bridge_ports bond0
+ bridge_stp off
+ bridge_fd 0
+
+----
+
+
+VLAN 802.1Q
+~~~~~~~~~~~
+
+A virtual LAN (VLAN) is a broadcast domain that is partitioned and
+isolated in the network at layer two. So it is possible to have
+multiple networks (4096) in a physical network, each independent of
+the other ones.
+
+Each VLAN network is identified by a number often called 'tag'.
+Network packages are then 'tagged' to identify which virtual network
+they belong to.
+
+
+VLAN for Guest Networks
+^^^^^^^^^^^^^^^^^^^^^^^
+
+{pve} supports this setup out of the box. You can specify the VLAN tag
+when you create a VM. The VLAN tag is part of the guest network
+confinuration. The networking layer supports differnet modes to
+implement VLANs, depending on the bridge configuration:
+
+* *VLAN awareness on the Linux bridge:*
+In this case, each guest's virtual network card is assigned to a VLAN tag,
+which is transparently supported by the Linux bridge.
+Trunk mode is also possible, but that makes the configuration
+in the guest necessary.
+
+* *"traditional" VLAN on the Linux bridge:*
+In contrast to the VLAN awareness method, this method is not transparent
+and creates a VLAN device with associated bridge for each VLAN.
+That is, if e.g. in our default network, a guest VLAN 5 is used
+to create eno1.5 and vmbr0v5, which remains until rebooting.
+
+* *Open vSwitch VLAN:*
+This mode uses the OVS VLAN feature.
+
+* *Guest configured VLAN:*
+VLANs are assigned inside the guest. In this case, the setup is
+completely done inside the guest and can not be influenced from the
+outside. The benefit is that you can use more than one VLAN on a
+single virtual NIC.
+
+
+VLAN on the Host
+^^^^^^^^^^^^^^^^
+
+To allow host communication with an isolated network. It is possible
+to apply VLAN tags to any network device (NIC, Bond, Bridge). In
+general, you should configure the VLAN on the interface with the least
+abstraction layers between itself and the physical NIC.
+
+For example, in a default configuration where you want to place
+the host management address on a separate VLAN.
+
+NOTE: In the examples we use the VLAN at bridge level to ensure the correct
+function of VLAN 5 in the guest network, but in combination with VLAN anwareness
+bridge this it will not work for guest network VLAN 5.
+The downside of this setup is more CPU usage.
+
+.Example: Use VLAN 5 for the {pve} management IP
+----
+auto lo
+iface lo inet loopback
+
+iface eno1 inet manual
+
+iface eno1.5 inet manual
+
+auto vmbr0v5
+iface vmbr0v5 inet static
+ address 10.10.10.2
+ netmask 255.255.255.0
+ gateway 10.10.10.1
+ bridge_ports eno1.5
+ bridge_stp off
+ bridge_fd 0
+
+auto vmbr0
+iface vmbr0 inet manual
+ bridge_ports eno1
+ bridge_stp off
+ bridge_fd 0
+
+----
+
+The next example is the same setup but a bond is used to
+make this network fail-safe.
+
+.Example: Use VLAN 5 with bond0 for the {pve} management IP
+----
+auto lo
+iface lo inet loopback
+
+iface eno1 inet manual
+
+iface eno2 inet manual
+
+auto bond0
+iface bond0 inet manual
+ slaves eno1 eno2
+ bond_miimon 100
+ bond_mode 802.3ad
+ bond_xmit_hash_policy layer2+3
+
+iface bond0.5 inet manual
+
+auto vmbr0v5
+iface vmbr0v5 inet static
+ address 10.10.10.2
+ netmask 255.255.255.0
+ gateway 10.10.10.1
+ bridge_ports bond0.5
+ bridge_stp off
+ bridge_fd 0
+
+auto vmbr0
+iface vmbr0 inet manual
+ bridge_ports bond0
+ bridge_stp off
+ bridge_fd 0
+