X-Git-Url: https://git.proxmox.com/?p=pve-docs.git;a=blobdiff_plain;f=vxlan-and-evpn.adoc;h=ec1bc0701ffd17a6f848c94108c5761cde5af020;hp=bd0bd72cf0bebfe1e3a90d814a7c93eae41926c5;hb=e13ba2ce9b2675dfdc480273f9c8587bbfec80d9;hpb=edcf21181052ee069c85bfd666c37d916ae5f2c5 diff --git a/vxlan-and-evpn.adoc b/vxlan-and-evpn.adoc index bd0bd72..ec1bc07 100644 --- a/vxlan-and-evpn.adoc +++ b/vxlan-and-evpn.adoc @@ -527,7 +527,7 @@ With this need, each vmbr bridge will be the gateway for the vm. Same vmbr on different node, will have same ip address and same mac address, to have working vm live migration and no network disruption. -VXLAN layer3 routing only work with FRR and non-aware bridge. +VXLAN layer3 routing only work with FRR and non-aware bridge. (vlan aware bridge support is buggy currently). asymmetric model @@ -535,29 +535,20 @@ asymmetric model This is the simplest mode. To get it work, all vxlan need to be defined on all nodes. -The asymmetric model allows routing and bridging on the VXLAN tunnel ingress, -but only bridging on the egress. -This results in bi-directional VXLAN traffic traveling on different VNIs +The asymmetric model allows routing and bridging on the VXLAN tunnel ingress, +but only bridging on the egress. +This results in bi-directional VXLAN traffic traveling on different VNIs in each direction (always the destination VNI) across the routed infrastructure. image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"] - -sysctl.conf tuning - ----- -#enable routing -net.ipv4.ip_forward=1 -net.ipv6.conf.all.forwarding=1 ----- - * node1 ---- auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.1 @@ -565,7 +556,7 @@ iface vmbr0 inet static bridge_ports eno1 bridge_stp off bridge_fd 0 - + auto vxlan2 iface vxlan2 inet manual vxlan-id 2 @@ -584,7 +575,9 @@ iface vmbr2 inet static bridge_ports vxlan2 bridge_stp off bridge_fd 0 - + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -604,6 +597,9 @@ iface vmbr3 inet static bridge_ports vxlan3 bridge_stp off bridge_fd 0 + ip-forward on + ip6-forward on + arp-accept on ---- @@ -620,7 +616,7 @@ router bgp 1234 address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate - advertise-all-vni + advertise-all-vni exit-address-family ! line vty @@ -634,7 +630,7 @@ line vty auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.2 @@ -642,7 +638,7 @@ iface vmbr0 inet static bridge_ports eno1 bridge_stp off bridge_fd 0 - + auto vxlan2 iface vxlan2 inet manual vxlan-id 2 @@ -661,6 +657,9 @@ iface vmbr2 inet static bridge_ports vxlan2 bridge_stp off bridge_fd 0 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 @@ -681,6 +680,9 @@ iface vmbr3 inet static bridge_ports vxlan3 bridge_stp off bridge_fd 0 + ip-forward on + ip6-forward on + arp-accept on ---- @@ -697,7 +699,7 @@ router bgp 1234 address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate - advertise-all-vni + advertise-all-vni exit-address-family ! line vty @@ -711,7 +713,7 @@ line vty auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.3 @@ -719,7 +721,7 @@ iface vmbr0 inet static bridge_ports eno1 bridge_stp off bridge_fd 0 - + auto vxlan2 iface vxlan2 inet manual vxlan-id 2 @@ -738,7 +740,9 @@ iface vmbr2 inet static bridge_ports vxlan2 bridge_stp off bridge_fd 0 - + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -749,7 +753,6 @@ iface vxlan3 inet manual bridge-unicast-flood off bridge-multicast-flood off - auto vmbr3 iface vmbr3 inet static address 10.0.3.254 @@ -758,6 +761,9 @@ iface vmbr3 inet static bridge_ports vxlan3 bridge_stp off bridge_fd 0 + ip-forward on + ip6-forward on + arp-accept on ---- @@ -774,7 +780,7 @@ router bgp 1234 address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate - advertise-all-vni + advertise-all-vni exit-address-family ! line vty @@ -786,29 +792,18 @@ symmetric model ^^^^^^^^^^^^^^^ With this model, you don't need to have all vxlan on all nodes. -This model will also be needed to route traffic to an external router. +This model will also be needed to route traffic to an external router. -The symmetric model routes and bridges on both the ingress and the egress leafs. -This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name. -However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI. -All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure, +The symmetric model routes and bridges on both the ingress and the egress leafs. +This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name. +However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI. +All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure, routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination. A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others. image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"] -sysctl.conf tuning - ----- -#enable routing -net.ipv4.ip_forward=1 -net.ipv6.conf.all.forwarding=1 -#disable reverse path filtering -net.ipv4.conf.default.rp_filter=0 -net.ipv4.conf.all.rp_filter=0 ----- - * node1 ---- @@ -819,7 +814,7 @@ iface vrf1 auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.1 @@ -846,6 +841,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -865,6 +863,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -882,7 +883,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:90 #must be different on each node vrf vrf1 ---- @@ -891,6 +891,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 @@ -905,18 +906,6 @@ router bgp 1234 advertise-all-vni exit-address-family ! -router bgp 1234 vrf vrf1 -! - bgp router-id 192.168.0.1 - ! - address-family ipv4 unicast - redistribute connected - exit-address-family - ! - address-family l2vpn evpn - advertise ipv4 unicast - exit-address-family -! line vty ! ---- @@ -932,7 +921,7 @@ iface vrf1 auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.2 @@ -959,6 +948,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -978,6 +970,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -995,7 +990,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:91 #must be different on each node vrf vrf1 ---- @@ -1005,6 +999,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.2 @@ -1019,18 +1014,6 @@ router bgp 1234 advertise-all-vni exit-address-family ! -router bgp 1234 vrf vrf1 -! - bgp router-id 192.168.0.2 - ! - address-family ipv4 unicast - redistribute connected - exit-address-family - ! - address-family l2vpn evpn - advertise ipv4 unicast - exit-address-family -! line vty ! ---- @@ -1046,7 +1029,7 @@ iface vrf1 auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.3 @@ -1073,6 +1056,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1092,6 +1078,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1109,7 +1098,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:92 #must be different on each node vrf vrf1 ---- @@ -1119,6 +1107,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.3 @@ -1133,31 +1122,18 @@ router bgp 1234 advertise-all-vni exit-address-family ! -router bgp 1234 vrf vrf1 -! - bgp router-id 192.168.0.3 - ! - address-family ipv4 unicast - redistribute connected - exit-address-family - ! - address-family l2vpn evpn - advertise ipv4 unicast - exit-address-family -! line vty ! ---- -VXLAN layer3 routing with anycast gateway + routing to outside with external router -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +VXLAN layer3 routing with anycast gateway + routing to outside with external router with static default gw +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Routing to outside need the symmetric model. 1 gateway node ^^^^^^^^^^^^^^ In this example, we'll use only 1 proxmox node as exit gateway. (node1) -This node have a simple default gw in the vrf to the external router (no bgp between router and node1) -and announce this default gw to other proxmox nodes. +This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1) *node1 @@ -1175,18 +1151,12 @@ auto vmbr0 iface vmbr0 inet static address 192.168.0.1 netmask 255.255.255.0 + gateway 192.168.0.254 bridge_ports eno1 bridge_stp off bridge_fd 0 - -auto eno2 -iface eno2 - address 172.16.0.1 - netmask 255.255.255.0 - vrf vrf1 - post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1 - #if you have multiple external routers, you can use ecmp balancing - #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1 + ip-forward on + ip6-forward on auto vxlan2 iface vxlan2 inet manual @@ -1206,6 +1176,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1225,6 +1198,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1241,7 +1217,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:90 #must be different on each node vrf vrf1 ---- @@ -1251,6 +1226,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 @@ -1259,6 +1235,14 @@ router bgp 1234 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! + address-family ipv4 unicast + import vrf vrf1 + exit-address-family + ! + address-family ipv6 unicast + import vrf vrf1 + exit-address-family + ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate @@ -1267,15 +1251,17 @@ router bgp 1234 ! router bgp 1234 vrf vrf1 ! - bgp router-id 172.16.0.1 - ! address-family ipv4 unicast redistribute connected - redistribute kernel !announce your default gw to all nodes + exit-address-family + ! + address-family ipv6 unicast + redistribute connected exit-address-family ! address-family l2vpn evpn - advertise ipv4 unicast + default-originate ipv4 + default-originate ipv6 exit-address-family ! line vty @@ -1293,7 +1279,7 @@ iface vrf1 auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.2 @@ -1320,6 +1306,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1339,6 +1328,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1356,7 +1348,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:91 #must be different on each node vrf vrf1 ---- @@ -1366,6 +1357,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.2 @@ -1380,18 +1372,6 @@ router bgp 1234 advertise-all-vni exit-address-family ! -router bgp 1234 vrf vrf1 -! - bgp router-id 192.168.0.2 - ! - address-family ipv4 unicast - redistribute connected - exit-address-family - ! - address-family l2vpn evpn - advertise ipv4 unicast - exit-address-family -! line vty ! ---- @@ -1407,7 +1387,7 @@ iface vrf1 auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.3 @@ -1434,6 +1414,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1453,6 +1436,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1470,7 +1456,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:92 #must be different on each node vrf vrf1 ---- @@ -1480,6 +1465,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.3 @@ -1494,18 +1480,6 @@ router bgp 1234 advertise-all-vni exit-address-family ! -router bgp 1234 vrf vrf1 -! - bgp router-id 192.168.0.3 - ! - address-family ipv4 unicast - redistribute connected - exit-address-family - ! - address-family l2vpn evpn - advertise ipv4 unicast - exit-address-family -! line vty ! ---- @@ -1513,12 +1487,22 @@ line vty multiple gateway nodes ^^^^^^^^^^^^^^^^^^^^^^ In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want) -All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1) -and announce this default gw. +All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1) +and announce this default gw in the vrf (default originate) The external router have ecmp routes to all proxmox nodes.(balancing). If the router send the packet to a wrong node (vm is not on this node), this node will route through vxlan the packet to final destination. +If you have multiple gateway nodes, disable rp_filter as packet could incoming in a 1 node, and outgoing +to another node. + +sysctl.conf tuning +----- +net.ipv4.conf.default.rp_filter=0 +net.ipv4.conf.all.rp_filter=0 +----- + + *node1 ---- @@ -1534,19 +1518,12 @@ auto vmbr0 iface vmbr0 inet static address 192.168.0.1 netmask 255.255.255.0 + gateway 192.168.0.254 bridge_ports eno1 bridge_stp off bridge_fd 0 - -auto eno2 -iface eno2 - address 172.16.0.1 - netmask 255.255.255.0 - vrf vrf1 - mtu 1550 - post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1 - #if you have multiple external routers, you can use ecmp balancing - #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1 + ip-forward on + ip6-forward on auto vxlan2 iface vxlan2 inet manual @@ -1566,6 +1543,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1585,6 +1565,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1601,7 +1584,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:90 #must be different on each node vrf vrf1 ---- @@ -1611,6 +1593,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 @@ -1619,6 +1602,14 @@ router bgp 1234 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! + address-family ipv4 unicast + import vrf vrf1 + exit-address-family + ! + address-family ipv6 unicast + import vrf vrf1 + exit-address-family + ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate @@ -1627,15 +1618,17 @@ router bgp 1234 ! router bgp 1234 vrf vrf1 ! - bgp router-id 172.16.0.1 - ! address-family ipv4 unicast redistribute connected - redistribute kernel !announce your default gw to all nodes + exit-address-family + ! + address-family ipv6 unicast + redistribute connected exit-address-family ! address-family l2vpn evpn - advertise ipv4 unicast + default-originate ipv4 + default-originate ipv6 exit-address-family ! line vty @@ -1653,24 +1646,17 @@ iface vrf1 auto eno1 iface eno1 inet manual mtu 1550 - + auto vmbr0 iface vmbr0 inet static address 192.168.0.2 netmask 255.255.255.0 + gateway 192.168.0.254 bridge_ports eno1 bridge_stp off bridge_fd 0 - -auto eno2 -iface eno2 - address 172.16.0.3 - netmask 255.255.255.0 - vrf vrf1 - mtu 1550 - post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1 - #if you have multiple external routers, you can use ecmp balancing - #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1 + ip-forward on + ip6-forward on auto vxlan2 iface vxlan2 inet manual @@ -1690,6 +1676,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1709,6 +1698,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1726,7 +1718,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:91 #must be different on each node vrf vrf1 ---- @@ -1736,6 +1727,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.2 @@ -1744,23 +1736,31 @@ router bgp 1234 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! + address-family ipv4 unicast + import vrf vrf1 + exit-address-family + ! + address-family ipv6 unicast + import vrf vrf1 + exit-address-family + ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! -router bgp 1234 vrf vrf1 -! - bgp router-id 172.16.0.2 - ! address-family ipv4 unicast redistribute connected - redistribute kernel !announce your default gw to all nodes + exit-address-family + ! + address-family ipv6 unicast + redistribute connected exit-address-family ! address-family l2vpn evpn - advertise ipv4 unicast + default-originate ipv4 + default-originate ipv6 exit-address-family ! line vty @@ -1783,19 +1783,12 @@ auto vmbr0 iface vmbr0 inet static address 192.168.0.3 netmask 255.255.255.0 + gateway 192.168.0.254 bridge_ports eno1 bridge_stp off bridge_fd 0 - -auto eno2 -iface eno2 - address 172.16.0.3 - netmask 255.255.255.0 - vrf vrf1 - mtu 1550 - post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1 - #if you have multiple external routers, you can use ecmp balancing - #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1 + ip-forward on + ip6-forward on auto vxlan2 iface vxlan2 inet manual @@ -1815,6 +1808,9 @@ iface vmbr2 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on auto vxlan3 iface vxlan3 inet manual @@ -1834,6 +1830,9 @@ iface vmbr3 inet static netmask 255.255.255.0 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3 vrf vrf1 + ip-forward on + ip6-forward on + arp-accept on #interconnect vxlan-vfr l3vni auto vxlan4000 @@ -1851,7 +1850,6 @@ iface vmbr4000 inet manual bridge_ports vxlan4000 bridge_stp off bridge_fd 0 - hwaddress 44:39:39:FF:40:92 #must be different on each node vrf vrf1 ---- @@ -1861,6 +1859,7 @@ frr.conf ---- vrf vrf1 vni 4000 + exit-vrf ! router bgp 1234 bgp router-id 192.168.0.3 @@ -1869,6 +1868,14 @@ router bgp 1234 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.2 remote-as 1234 ! + address-family ipv4 unicast + import vrf vrf1 + exit-address-family + ! + address-family ipv6 unicast + import vrf vrf1 + exit-address-family + ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate @@ -1877,15 +1884,17 @@ router bgp 1234 ! router bgp 1234 vrf vrf1 ! - bgp router-id 172.16.0.3 - ! address-family ipv4 unicast redistribute connected - redistribute kernel !announce your default gw to all nodes + exit-address-family + ! + address-family ipv6 unicast + redistribute connected exit-address-family ! address-family l2vpn evpn - advertise ipv4 unicast + default-originate ipv4 + default-originate ipv6 exit-address-family ! line vty @@ -1895,41 +1904,204 @@ line vty Note ^^^^ -If your external router don't support ecmp to reach multiple proxmox nodes, -you can setup an HA floating vip on proxmox nodes with vrrp +If your external router doesn't support 'ECMP static routes' to reach multiple +{pve} nodes, you can setup an HA floating vip on proxmox nodes by using the +Virtual Router Redundancy Protocol (VRRP). -I this example, we will setup an floating 172.16.0.10 ip on node1 and node2. -Node1 is the primary and failover to node2 in case of failure. +In this example, we will setup an floating 192.168.0.10 IP on node1 and node2. +Node1 is the primary with failover to node2 in case of outage. +This setup currently needs 'vrrpd' package (`apt install vrrpd`). +#TODO : It should be possible to do it with frr directly with last version. * node1 ---- -auto eno2 -iface eno2 - address 172.16.0.1 - netmask 255.255.255.0 - vrf vrf1 - mtu 1550 - post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1 - vrrp-id 1 - vrrp-priority 1 - vrrp-virtual-ip 172.16.0.10 +auto vmbr0 +iface vmbr0 inet static + address 192.168.0.1 + netmask 255.255.255.0 + gateway 192.168.0.254 + bridge_ports eno1 + bridge_stp off + bridge_fd 0 + vrrp-id 1 + vrrp-priority 1 + vrrp-virtual-ip 192.168.0.10 ---- * node2 ---- -auto eno2 -iface eno2 - address 172.16.0.2 - netmask 255.255.255.0 - mtu 1550 - vrf vrf1 - post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1 - vrrp-id 1 - vrrp-priority 2 - vrrp-virtual-ip 172.16.0.10 +auto vmbr0 +iface vmbr0 inet static + address 192.168.0.2 + netmask 255.255.255.0 + gateway 192.168.0.254 + bridge_ports eno1 + bridge_stp off + bridge_fd 0 + vrrp-id 1 + vrrp-priority 2 + vrrp-virtual-ip 192.168.0.10 +---- + + + +gateway node(s) with a upstream bgp router +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Setup is almost the same than with a static gateway, but we'll connect to an upstream bgp router. + +example with node1 as gateway (192.168.0.1) for evpn-bgp, and an upstream bgp router (running frr too) 192.168.0.254. + +* node1 + +frr.conf +---- +vrf vrf1 + vni 4000 + exit-vrf +! +router bgp 1234 + bgp router-id 192.168.0.1 + no bgp default ipv4-unicast + coalesce-time 1000 + neighbor 192.168.0.2 remote-as 1234 + neighbor 192.168.0.3 remote-as 1234 + neighbor 192.168.0.254 remote-as external + ! + address-family ipv4 unicast + import vrf vrf1 + neighbor 192.168.0.254 activate + exit-address-family + ! + address-family ipv6 unicast + import vrf vrf1 + neighbor 192.168.0.254 activate + exit-address-family + ! + address-family l2vpn evpn + neighbor 192.168.0.1 activate + neighbor 192.168.0.2 activate + neighbor 192.168.0.254 activate + advertise-all-vni + exit-address-family +! +router bgp 1234 vrf vrf1 +! + address-family ipv4 unicast + redistribute connected + exit-address-family + ! + address-family ipv6 unicast + redistribute connected + exit-address-family + ! + address-family l2vpn evpn + default-originate ipv4 + default-originate ipv6 + exit-address-family +! +line vty +! +---- + +* bgp router + +frr.conf ---- +ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24 +ip prefix-list NO32 seq 20 deny any +! +router bgp 25253 + bgp router-id 192.168.0.254 + bgp bestpath as-path multipath-relax + neighbor 192.168.0.1 remote-as external + neighbor 192.168.0.1 capability extended-nexthop + ! + address-family ipv4 unicast + neighbor 192.168.0.1 default-originate + neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn + exit-address-family + ! + address-family ipv6 unicast + neighbor 192.168.0.1 default-originate + neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn + exit-address-family + ! +! +--- + +Route Reflectors +^^^^^^^^^^^^^^^^ +If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want +to avoid that all node peers with each others nodes. +For this, you can create dedicated route reflectors (RR) servers. As a RR is a +single point of failure, a minimum of two servers acting as an RR is highly +recommended for redundancy. + +Below is an example of configuration with 'frr', with `rrserver1 +(192.168.0.200)' and `rrserver2 (192.168.0.201)`. +rrserver1 +---- +router bgp 1234 + bgp router-id 192.168.0.200 + bgp cluster-id 1.1.1.1 #cluster-id must be the same on each route reflector + bgp log-neighbor-changes + no bgp default ipv4-unicast + neighbor fabric peer-group + neighbor fabric remote-as 1234 + neighbor fabric capability extended-nexthop + neighbor fabric update-source 192.168.0.200 + bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range + ! + address-family l2vpn evpn + neighbor fabric activate + neighbor fabric route-reflector-client + neighbor fabric allowas-in + exit-address-family + ! + exit +! +--- +rrserver2 +---- +router bgp 1234 + bgp router-id 192.168.0.201 + bgp cluster-id 1.1.1.1 + bgp log-neighbor-changes + no bgp default ipv4-unicast + neighbor fabric peer-group + neighbor fabric remote-as 1234 + neighbor fabric capability extended-nexthop + neighbor fabric update-source 192.168.0.201 + bgp listen range 192.168.0.0/24 peer-group fabric + ! + address-family l2vpn evpn + neighbor fabric activate + neighbor fabric route-reflector-client + neighbor fabric allowas-in + exit-address-family + ! + exit +! +--- + +proxmoxnode(s) +---- +router bgp 1234 + bgp router-id 192.168.0.x + no bgp default ipv4-unicast + coalesce-time 1000 + neighbor 192.168.0.200 remote-as 1234 + neighbor 192.168.0.201 remote-as 1234 + ! + address-family l2vpn evpn + neighbor 192.168.0.200 activate + neighbor 192.168.0.201 activate + advertise-all-vni + exit-address-family +! +----