4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
27 This scenario relies in head end replication, meaning that end host in case
28 of not having any entry for the destination MAC address will send out an ARP
29 to other devices / VTEPs in the VXLAN network.
30 This is done by sending the request to the VXLAN multicast group,
31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
38 iface eno1 inet manual
41 iface vmbr0 inet static
49 iface vxlan2 inet manual
50 vxlan-svcnodeip 225.20.1.1
54 iface vmbr2 inet manual
60 iface vxlan3 inet manual
61 vxlan-svcnodeip 225.20.1.1
65 iface vmbr3 inet manual
76 iface eno1 inet manual
79 iface vmbr0 inet static
87 iface vxlan2 inet manual
88 vxlan-svcnodeip 225.20.1.1
92 iface vmbr2 inet manual
99 iface vxlan3 inet manual
100 vxlan-svcnodeip 225.20.1.1
104 iface vmbr3 inet manual
115 iface eno1 inet manual
118 iface vmbr0 inet static
120 netmask 255.255.255.0
126 iface vxlan2 inet manual
127 vxlan-svcnodeip 225.20.1.1
131 iface vmbr2 inet manual
138 iface vxlan3 inet manual
139 vxlan-svcnodeip 225.20.1.1
143 iface vmbr3 inet manual
153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
154 The VXLAN is defined without a remote multicast group.
155 Instead, all the remote VTEPs are associated with the all-zero address:
156 a BUM frame will be duplicated to all these destinations.
157 The VXLAN device will still learn remote addresses automatically using source-address learning.
163 iface eno1 inet manual
166 iface vmbr0 inet static
168 netmask 255.255.255.0
175 iface vxlan2 inet manual
176 vxlan_remoteip 192.168.0.2
177 vxlan_remoteip 192.168.0.3
181 iface vmbr2 inet manual
188 iface vxlan2 inet manual
189 vxlan_remoteip 192.168.0.2
190 vxlan_remoteip 192.168.0.3
194 iface vmbr3 inet manual
205 iface eno1 inet manual
208 iface vmbr0 inet static
210 netmask 255.255.255.0
216 iface vxlan2 inet manual
217 vxlan_remoteip 192.168.0.1
218 vxlan_remoteip 192.168.0.3
223 iface vmbr2 inet manual
229 iface vxlan2 inet manual
230 vxlan_remoteip 192.168.0.1
231 vxlan_remoteip 192.168.0.3
235 iface vmbr3 inet manual
246 iface eno1 inet manual
249 iface vmbr0 inet static
251 netmask 255.255.255.0
257 iface vxlan2 inet manual
258 vxlan_remoteip 192.168.0.2
259 vxlan_remoteip 192.168.0.3
264 iface vmbr2 inet manual
270 iface vxlan2 inet manual
271 vxlan_remoteip 192.168.0.2
272 vxlan_remoteip 192.168.0.3
276 iface vmbr3 inet manual
286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
289 The control plane used here is FRR, a bgp routing software.
290 Each node in the proxmox cluster peer with each others nodes.
291 For bigger networks, or multiple proxmox clusters,
292 it's possible to use external bgp route reflector servers.
298 iface eno1 inet manual
301 iface vmbr0 inet static
303 netmask 255.255.255.0
309 iface vxlan2 inet manual
310 vxlan-local-tunnelip 192.168.0.1
312 bridge-arp-nd-suppress on
313 bridge-unicast-flood off
314 bridge-multicast-flood off
318 iface vmbr2 inet manual
325 iface vxlan3 inet manual
326 vxlan-local-tunnelip 192.168.0.1
328 bridge-arp-nd-suppress on
329 bridge-unicast-flood off
330 bridge-multicast-flood off
334 iface vmbr3 inet manual
345 no bgp default ipv4-unicast
347 neighbor 192.168.0.2 remote-as 1234
348 neighbor 192.168.0.3 remote-as 1234
350 address-family l2vpn evpn
351 neighbor 192.168.0.2 activate
352 neighbor 192.168.0.3 activate
365 iface eno1 inet manual
368 iface vmbr0 inet static
370 netmask 255.255.255.0
376 iface vxlan2 inet manual
377 vxlan-local-tunnelip 192.168.0.2
379 bridge-arp-nd-suppress on
380 bridge-unicast-flood off
381 bridge-multicast-flood off
385 iface vmbr2 inet manual
391 iface vxlan3 inet manual
392 vxlan-local-tunnelip 192.168.0.2
394 bridge-arp-nd-suppress on
395 bridge-unicast-flood off
396 bridge-multicast-flood off
400 iface vmbr3 inet manual
411 no bgp default ipv4-unicast
413 neighbor 192.168.0.1 remote-as 1234
414 neighbor 192.168.0.3 remote-as 1234
416 address-family l2vpn evpn
417 neighbor 192.168.0.1 activate
418 neighbor 192.168.0.3 activate
431 iface eno1 inet manual
434 iface vmbr0 inet static
436 netmask 255.255.255.0
442 iface vxlan2 inet manual
443 vxlan-local-tunnelip 192.168.0.3
445 bridge-arp-nd-suppress on
446 bridge-unicast-flood off
447 bridge-multicast-flood off
451 iface vmbr2 inet manual
457 iface vxlan3 inet manual
458 vxlan-local-tunnelip 192.168.0.3
460 bridge-arp-nd-suppress on
461 bridge-unicast-flood off
462 bridge-multicast-flood off
466 iface vmbr3 inet manual
478 no bgp default ipv4-unicast
480 neighbor 192.168.0.1 remote-as 1234
481 neighbor 192.168.0.2 remote-as 1234
483 address-family l2vpn evpn
484 neighbor 192.168.0.1 activate
485 neighbor 192.168.0.2 activate
493 VXLAN layer3 routing with anycast gateway
494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
496 With this need, each vmbr bridge will be the gateway for the vm.
497 Same vmbr on different node, will have same ip address and same mac address,
498 to have working vm live migration and no network disruption.
500 VXLAN layer3 routing only work with FRR and non-aware bridge.
501 (vlan aware bridge support is buggy currently).
506 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
508 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
509 but only bridging on the egress.
510 This results in bi-directional VXLAN traffic traveling on different VNIs
511 in each direction (always the destination VNI) across the routed infrastructure.
513 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
520 net.ipv4.ip_forward=1
521 net.ipv6.conf.all.forwarding=1
528 iface eno1 inet manual
531 iface vmbr0 inet static
533 netmask 255.255.255.0
539 iface vxlan2 inet manual
540 vxlan-local-tunnelip 192.168.0.1
542 bridge-arp-nd-suppress on
543 bridge-unicast-flood off
544 bridge-multicast-flood off
548 iface vmbr2 inet static
550 netmask 255.255.255.0
551 hwaddress 44:39:39:FF:40:94
558 iface vxlan3 inet manual
559 vxlan-local-tunnelip 192.168.0.1
561 bridge-arp-nd-suppress on
562 bridge-unicast-flood off
563 bridge-multicast-flood off
567 iface vmbr3 inet static
569 netmask 255.255.255.0
570 hwaddress 44:39:39:FF:40:94
581 bgp router-id 192.168.0.1
582 no bgp default ipv4-unicast
584 neighbor 192.168.0.2 remote-as 1234
585 neighbor 192.168.0.3 remote-as 1234
587 address-family l2vpn evpn
588 neighbor 192.168.0.2 activate
589 neighbor 192.168.0.3 activate
602 iface eno1 inet manual
605 iface vmbr0 inet static
607 netmask 255.255.255.0
613 iface vxlan2 inet manual
614 vxlan-local-tunnelip 192.168.0.2
616 bridge-arp-nd-suppress on
617 bridge-unicast-flood off
618 bridge-multicast-flood off
622 iface vmbr2 inet static
624 netmask 255.255.255.0
625 hwaddress 44:39:39:FF:40:94
632 iface vxlan3 inet manual
633 vxlan-local-tunnelip 192.168.0.2
635 bridge-arp-nd-suppress on
636 bridge-unicast-flood off
637 bridge-multicast-flood off
641 iface vmbr3 inet static
643 netmask 255.255.255.0
644 hwaddress 44:39:39:FF:40:94
655 bgp router-id 192.168.0.2
656 no bgp default ipv4-unicast
658 neighbor 192.168.0.1 remote-as 1234
659 neighbor 192.168.0.3 remote-as 1234
661 address-family l2vpn evpn
662 neighbor 192.168.0.1 activate
663 neighbor 192.168.0.3 activate
676 iface eno1 inet manual
679 iface vmbr0 inet static
681 netmask 255.255.255.0
687 iface vxlan2 inet manual
688 vxlan-local-tunnelip 192.168.0.3
690 bridge-arp-nd-suppress on
691 bridge-unicast-flood off
692 bridge-multicast-flood off
696 iface vmbr2 inet static
698 netmask 255.255.255.0
699 hwaddress 44:39:39:FF:40:94
706 iface vxlan3 inet manual
707 vxlan-local-tunnelip 192.168.0.3
709 bridge-arp-nd-suppress on
710 bridge-unicast-flood off
711 bridge-multicast-flood off
715 iface vmbr3 inet static
717 netmask 255.255.255.0
718 hwaddress 44:39:39:FF:40:94
729 bgp router-id 192.168.0.3
730 no bgp default ipv4-unicast
732 neighbor 192.168.0.1 remote-as 1234
733 neighbor 192.168.0.2 remote-as 1234
735 address-family l2vpn evpn
736 neighbor 192.168.0.1 activate
737 neighbor 192.168.0.2 activate
749 With this model, you don't need to have all vxlan on all nodes.
750 This model will also be needed to route traffic to an external router.
752 The symmetric model routes and bridges on both the ingress and the egress leafs.
753 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
754 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
755 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
756 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
758 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
760 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
766 net.ipv4.ip_forward=1
767 net.ipv6.conf.all.forwarding=1
768 #disable reverse path filtering
769 net.ipv4.conf.default.rp_filter=0
770 net.ipv4.conf.all.rp_filter=0
771 #allow frr to work with vrf (kernel >4.14 bug)
772 net.ipv4.tcp_l3mdev_accept=1
783 iface eno1 inet manual
786 iface vmbr0 inet static
788 netmask 255.255.255.0
794 iface vxlan2 inet manual
795 vxlan-local-tunnelip 192.168.0.1
797 bridge-arp-nd-suppress on
798 bridge-unicast-flood off
799 bridge-multicast-flood off
802 iface vmbr2 inet static
807 netmask 255.255.255.0
808 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
812 iface vxlan3 inet manual
813 vxlan-local-tunnelip 192.168.0.1
815 bridge-arp-nd-suppress on
816 bridge-unicast-flood off
817 bridge-multicast-flood off
820 iface vmbr3 inet static
825 netmask 255.255.255.0
826 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
829 #interconnect vxlan-vfr l3vni
831 iface vxlan4000 inet manual
832 vxlan-local-tunnelip 192.168.0.1
834 bridge-arp-nd-suppress on
835 bridge-unicast-flood off
836 bridge-multicast-flood off
840 iface vmbr4000 inet manual
841 bridge_ports vxlan4000
844 hwaddress 44:39:39:FF:40:90 #must be different on each node
855 bgp router-id 192.168.0.1
856 no bgp default ipv4-unicast
858 neighbor 192.168.0.2 remote-as 1234
859 neighbor 192.168.0.3 remote-as 1234
861 address-family l2vpn evpn
862 neighbor 192.168.0.2 activate
863 neighbor 192.168.0.3 activate
867 router bgp 1234 vrf vrf1
869 bgp router-id 192.168.0.1
871 address-family ipv4 unicast
872 redistribute connected
875 address-family l2vpn evpn
876 advertise ipv4 unicast
892 iface eno1 inet manual
895 iface vmbr0 inet static
897 netmask 255.255.255.0
903 iface vxlan2 inet manual
904 vxlan-local-tunnelip 192.168.0.2
906 bridge-arp-nd-suppress on
907 bridge-unicast-flood off
908 bridge-multicast-flood off
911 iface vmbr2 inet static
916 netmask 255.255.255.0
917 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
921 iface vxlan3 inet manual
922 vxlan-local-tunnelip 192.168.0.2
924 bridge-arp-nd-suppress on
925 bridge-unicast-flood off
926 bridge-multicast-flood off
929 iface vmbr3 inet static
934 netmask 255.255.255.0
935 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
938 #interconnect vxlan-vfr l3vni
940 iface vxlan4000 inet manual
941 vxlan-local-tunnelip 192.168.0.2
943 bridge-arp-nd-suppress on
944 bridge-unicast-flood off
945 bridge-multicast-flood off
949 iface vmbr4000 inet manual
950 bridge_ports vxlan4000
953 hwaddress 44:39:39:FF:40:91 #must be different on each node
965 bgp router-id 192.168.0.2
966 no bgp default ipv4-unicast
968 neighbor 192.168.0.1 remote-as 1234
969 neighbor 192.168.0.3 remote-as 1234
971 address-family l2vpn evpn
972 neighbor 192.168.0.1 activate
973 neighbor 192.168.0.3 activate
977 router bgp 1234 vrf vrf1
979 bgp router-id 192.168.0.2
981 address-family ipv4 unicast
982 redistribute connected
985 address-family l2vpn evpn
986 advertise ipv4 unicast
1002 iface eno1 inet manual
1005 iface vmbr0 inet static
1007 netmask 255.255.255.0
1013 iface vxlan2 inet manual
1014 vxlan-local-tunnelip 192.168.0.3
1016 bridge-arp-nd-suppress on
1017 bridge-unicast-flood off
1018 bridge-multicast-flood off
1021 iface vmbr2 inet static
1026 netmask 255.255.255.0
1027 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1031 iface vxlan3 inet manual
1032 vxlan-local-tunnelip 192.168.0.3
1034 bridge-arp-nd-suppress on
1035 bridge-unicast-flood off
1036 bridge-multicast-flood off
1039 iface vmbr3 inet static
1044 netmask 255.255.255.0
1045 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1048 #interconnect vxlan-vfr l3vni
1050 iface vxlan4000 inet manual
1051 vxlan-local-tunnelip 192.168.0.3
1053 bridge-arp-nd-suppress on
1054 bridge-unicast-flood off
1055 bridge-multicast-flood off
1059 iface vmbr4000 inet manual
1060 bridge_ports vxlan4000
1063 hwaddress 44:39:39:FF:40:92 #must be different on each node
1075 bgp router-id 192.168.0.3
1076 no bgp default ipv4-unicast
1078 neighbor 192.168.0.1 remote-as 1234
1079 neighbor 192.168.0.2 remote-as 1234
1081 address-family l2vpn evpn
1082 neighbor 192.168.0.1 activate
1083 neighbor 192.168.0.2 activate
1087 router bgp 1234 vrf vrf1
1089 bgp router-id 192.168.0.3
1091 address-family ipv4 unicast
1092 redistribute connected
1095 address-family l2vpn evpn
1096 advertise ipv4 unicast