physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
For BUM traffic (broadcast / unknown unicast traffic, multicast),
-we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
+we have 3 different VXLAN setup modes : multicast, unicast, bgp-evpn
image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
----
router bgp 1234
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.2 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
----
router bgp 1234
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
----
router bgp 1234
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.2 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.1
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.2 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.2
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.3
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.2 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.1
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.2 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.2
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.3
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.2 remote-as 1234
!
----
-VXLAN layer3 routing with anycast gateway + routing to outside with external router
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+VXLAN layer3 routing with anycast gateway + routing to outside with external router with static default gw
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Routing to outside need the symmetric model.
1 gateway node
router bgp 1234
bgp router-id 192.168.0.1
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.2 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.2
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.3
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.2 remote-as 1234
If the router send the packet to a wrong node (vm is not on this node), this node will route through
vxlan the packet to final destination.
+If you have multiple gateway nodes, disable rp_filter as packet could incoming in a 1 node, and outgoing
+to another node.
+
+sysctl.conf tuning
+-----
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0
+-----
+
+
*node1
----
router bgp 1234
bgp router-id 192.168.0.1
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.2 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.2
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.3 remote-as 1234
router bgp 1234
bgp router-id 192.168.0.3
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.1 remote-as 1234
neighbor 192.168.0.2 remote-as 1234
----
+
+gateway node(s) with a upstream bgp router
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Setup is almost the same than with a static gateway, but we'll connect to an upstream bgp router.
+
+example with node1 as gateway (192.168.0.1) for evpn-bgp, and an upstream bgp router (running frr too) 192.168.0.254.
+
+* node1
+
+frr.conf
+----
+vrf vrf1
+ vni 4000
+ exit-vrf
+!
+router bgp 1234
+ bgp router-id 192.168.0.1
+ no bgp default ipv4-unicast
+ coalesce-time 1000
+ neighbor 192.168.0.2 remote-as 1234
+ neighbor 192.168.0.3 remote-as 1234
+ neighbor 192.168.0.254 remote-as external
+ !
+ address-family ipv4 unicast
+ import vrf vrf1
+ neighbor 192.168.0.254 activate
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ import vrf vrf1
+ neighbor 192.168.0.254 activate
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ neighbor 192.168.0.1 activate
+ neighbor 192.168.0.2 activate
+ neighbor 192.168.0.254 activate
+ advertise-all-vni
+ exit-address-family
+!
+router bgp 1234 vrf vrf1
+!
+ address-family ipv4 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ redistribute connected
+ exit-address-family
+ !
+ address-family l2vpn evpn
+ default-originate ipv4
+ default-originate ipv6
+ exit-address-family
+!
+line vty
+!
+----
+
+* bgp router
+
+frr.conf
+----
+ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24
+ip prefix-list NO32 seq 20 deny any
+!
+router bgp 25253
+ bgp router-id 192.168.0.254
+ bgp bestpath as-path multipath-relax
+ neighbor 192.168.0.1 remote-as external
+ neighbor 192.168.0.1 capability extended-nexthop
+ !
+ address-family ipv4 unicast
+ neighbor 192.168.0.1 default-originate
+ neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
+ exit-address-family
+ !
+ address-family ipv6 unicast
+ neighbor 192.168.0.1 default-originate
+ neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
+ exit-address-family
+ !
+!
+---
+
Route Reflectors
^^^^^^^^^^^^^^^^
If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
bgp cluster-id 1.1.1.1 #cluster-id must be the same on each route reflector
bgp log-neighbor-changes
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
neighbor fabric peer-group
neighbor fabric remote-as 1234
neighbor fabric capability extended-nexthop
bgp cluster-id 1.1.1.1
bgp log-neighbor-changes
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
neighbor fabric peer-group
neighbor fabric remote-as 1234
neighbor fabric capability extended-nexthop
router bgp 1234
bgp router-id 192.168.0.x
no bgp default ipv4-unicast
- no bgp default ipv6-unicast
coalesce-time 1000
neighbor 192.168.0.200 remote-as 1234
neighbor 192.168.0.201 remote-as 1234
exit-address-family
!
----
-
-#TODO : Documentation with bgp upstream router.