physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
For BUM traffic (broadcast / unknown unicast traffic, multicast),
-we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
+we have 3 different VXLAN setup modes : multicast, unicast, bgp-evpn
image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
-
-sysctl.conf tuning
-
-----
-#enable routing
-net.ipv4.ip_forward=1
-net.ipv6.conf.all.forwarding=1
-----
-
* node1
----
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
-
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
+ arp-accept on
----
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
+ arp-accept on
----
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
-
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
bridge-unicast-flood off
bridge-multicast-flood off
-
auto vmbr3
iface vmbr3 inet static
address 10.0.3.254
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
+ arp-accept on
----
image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
-sysctl.conf tuning
-
-----
-#enable routing
-net.ipv4.ip_forward=1
-net.ipv6.conf.all.forwarding=1
-----
-
* node1
----
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
!
----
-VXLAN layer3 routing with anycast gateway + routing to outside with external router
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+VXLAN layer3 routing with anycast gateway + routing to outside with external router with static default gw
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Routing to outside need the symmetric model.
1 gateway node
bridge_ports eno1
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
auto vxlan2
iface vxlan2 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
import vrf vrf1
exit-address-family
!
+ address-family ipv6 unicast
+ import vrf vrf1
+ exit-address-family
+ !
address-family l2vpn evpn
neighbor 192.168.0.2 activate
neighbor 192.168.0.3 activate
!
router bgp 1234 vrf vrf1
!
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ redistribute connected
+ exit-address-family
+ !
address-family l2vpn evpn
default-originate ipv4
+ default-originate ipv6
exit-address-family
!
line vty
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
If the router send the packet to a wrong node (vm is not on this node), this node will route through
vxlan the packet to final destination.
+If you have multiple gateway nodes, disable rp_filter as packet could incoming in a 1 node, and outgoing
+to another node.
+
+sysctl.conf tuning
+-----
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0
+-----
+
+
*node1
----
bridge_ports eno1
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
auto vxlan2
iface vxlan2 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
import vrf vrf1
exit-address-family
!
+ address-family ipv6 unicast
+ import vrf vrf1
+ exit-address-family
+ !
address-family l2vpn evpn
neighbor 192.168.0.2 activate
neighbor 192.168.0.3 activate
!
router bgp 1234 vrf vrf1
!
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ redistribute connected
+ exit-address-family
+ !
address-family l2vpn evpn
default-originate ipv4
+ default-originate ipv6
exit-address-family
!
line vty
bridge_ports eno1
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
auto vxlan2
iface vxlan2 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
import vrf vrf1
exit-address-family
!
+ address-family ipv6 unicast
+ import vrf vrf1
+ exit-address-family
+ !
address-family l2vpn evpn
neighbor 192.168.0.1 activate
neighbor 192.168.0.3 activate
advertise-all-vni
exit-address-family
!
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ redistribute connected
+ exit-address-family
+ !
address-family l2vpn evpn
default-originate ipv4
+ default-originate ipv6
exit-address-family
!
line vty
bridge_ports eno1
bridge_stp off
bridge_fd 0
+ ip-forward on
+ ip6-forward on
auto vxlan2
iface vxlan2 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
auto vxlan3
iface vxlan3 inet manual
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
+ ip-forward on
+ ip6-forward on
+ arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
import vrf vrf1
exit-address-family
!
+ address-family ipv6 unicast
+ import vrf vrf1
+ exit-address-family
+ !
address-family l2vpn evpn
neighbor 192.168.0.1 activate
neighbor 192.168.0.2 activate
!
router bgp 1234 vrf vrf1
!
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ redistribute connected
+ exit-address-family
+ !
address-family l2vpn evpn
default-originate ipv4
+ default-originate ipv6
exit-address-family
!
line vty
Note
^^^^
-If your external router don't support ecmp static route to reach multiple proxmox nodes,
-you can setup an HA floating vip on proxmox nodes with vrrp
+If your external router doesn't support 'ECMP static routes' to reach multiple
+{pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
+Virtual Router Redundancy Protocol (VRRP).
-In this example, we will setup an floating 192.168.0.10 ip on node1 and node2.
-Node1 is the primary and failover to node2 in case of failure.
+In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
+Node1 is the primary with failover to node2 in case of outage.
-This setup need vrrpd package (apt install vrrpd).
+This setup currently needs 'vrrpd' package (`apt install vrrpd`).
#TODO : It should be possible to do it with frr directly with last version.
* node1
----
-route reflectors
+
+gateway node(s) with a upstream bgp router
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Setup is almost the same than with a static gateway, but we'll connect to an upstream bgp router.
+
+example with node1 as gateway (192.168.0.1) for evpn-bgp, and an upstream bgp router (running frr too) 192.168.0.254.
+
+* node1
+
+frr.conf
+----
+vrf vrf1
+ vni 4000
+ exit-vrf
+!
+router bgp 1234
+ bgp router-id 192.168.0.1
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.2 remote-as 1234
+ neighbor 192.168.0.3 remote-as 1234
+ neighbor 192.168.0.254 remote-as external
+ !
+ address-family ipv4 unicast
+ import vrf vrf1
+ neighbor 192.168.0.254 activate
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ import vrf vrf1
+ neighbor 192.168.0.254 activate
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.1 activate
+ neighbor 192.168.0.2 activate
+ neighbor 192.168.0.254 activate
+ advertise-all-vni
+ exit-address-family
+!
+router bgp 1234 vrf vrf1
+!
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ default-originate ipv4
+ default-originate ipv6
+ exit-address-family
+!
+line vty
+!
+----
+
+* bgp router
+
+frr.conf
+----
+ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24
+ip prefix-list NO32 seq 20 deny any
+!
+router bgp 25253
+ bgp router-id 192.168.0.254
+ bgp bestpath as-path multipath-relax
+ neighbor 192.168.0.1 remote-as external
+ neighbor 192.168.0.1 capability extended-nexthop
+ !
+ address-family ipv4 unicast
+ neighbor 192.168.0.1 default-originate
+ neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ neighbor 192.168.0.1 default-originate
+ neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
+ exit-address-family
+ !
+!
+---
+
+Route Reflectors
^^^^^^^^^^^^^^^^
-If you have a lot of proxmox nodes, or multiple proxmox clusters,
-maybe do you want to avoid that each node peer with each others nodes.
-For this, you can create dedicated route reflectors servers. (Minimum 2 servers for redundancy).
-Here an example of configuration with frr, with rrserver1 (192.168.0.200) and rrserver2 (192.168.0.201).
+If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
+to avoid that all node peers with each others nodes.
+For this, you can create dedicated route reflectors (RR) servers. As a RR is a
+single point of failure, a minimum of two servers acting as an RR is highly
+recommended for redundancy.
+Below is an example of configuration with 'frr', with `rrserver1
+(192.168.0.200)' and `rrserver2 (192.168.0.201)`.
rrserver1
----
neighbor 192.168.0.200 remote-as 1234
neighbor 192.168.0.201 remote-as 1234
!
- address-family ipv4 unicast
- import vrf vrf1
- exit-address-family
- !
address-family l2vpn evpn
neighbor 192.168.0.200 activate
neighbor 192.168.0.201 activate
exit-address-family
!
----
-
-#TODO : Documentation with bgp upstream router.
projects / pve-docs.git / blobdiff