4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
27 This scenario relies in head end replication, meaning that end host in case
28 of not having any entry for the destination MAC address will send out an ARP
29 to other devices / VTEPs in the VXLAN network.
30 This is done by sending the request to the VXLAN multicast group,
31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
38 iface eno1 inet manual
41 iface vmbr0 inet static
49 iface vxlan2 inet manual
50 vxlan-svcnodeip 225.20.1.1
54 iface vmbr2 inet manual
60 iface vxlan3 inet manual
61 vxlan-svcnodeip 225.20.1.1
65 iface vmbr3 inet manual
76 iface eno1 inet manual
79 iface vmbr0 inet static
87 iface vxlan2 inet manual
88 vxlan-svcnodeip 225.20.1.1
92 iface vmbr2 inet manual
99 iface vxlan3 inet manual
100 vxlan-svcnodeip 225.20.1.1
104 iface vmbr3 inet manual
115 iface eno1 inet manual
118 iface vmbr0 inet static
120 netmask 255.255.255.0
126 iface vxlan2 inet manual
127 vxlan-svcnodeip 225.20.1.1
131 iface vmbr2 inet manual
138 iface vxlan3 inet manual
139 vxlan-svcnodeip 225.20.1.1
143 iface vmbr3 inet manual
153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
154 The VXLAN is defined without a remote multicast group.
155 Instead, all the remote VTEPs are associated with the all-zero address:
156 a BUM frame will be duplicated to all these destinations.
157 The VXLAN device will still learn remote addresses automatically using source-address learning.
163 iface eno1 inet manual
166 iface vmbr0 inet static
168 netmask 255.255.255.0
175 iface vxlan2 inet manual
176 vxlan_remoteip 192.168.0.2
177 vxlan_remoteip 192.168.0.3
181 iface vmbr2 inet manual
188 iface vxlan2 inet manual
189 vxlan_remoteip 192.168.0.2
190 vxlan_remoteip 192.168.0.3
194 iface vmbr3 inet manual
205 iface eno1 inet manual
208 iface vmbr0 inet static
210 netmask 255.255.255.0
216 iface vxlan2 inet manual
217 vxlan_remoteip 192.168.0.1
218 vxlan_remoteip 192.168.0.3
223 iface vmbr2 inet manual
229 iface vxlan2 inet manual
230 vxlan_remoteip 192.168.0.1
231 vxlan_remoteip 192.168.0.3
235 iface vmbr3 inet manual
246 iface eno1 inet manual
249 iface vmbr0 inet static
251 netmask 255.255.255.0
257 iface vxlan2 inet manual
258 vxlan_remoteip 192.168.0.2
259 vxlan_remoteip 192.168.0.3
264 iface vmbr2 inet manual
270 iface vxlan2 inet manual
271 vxlan_remoteip 192.168.0.2
272 vxlan_remoteip 192.168.0.3
276 iface vmbr3 inet manual
286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
289 The control plane used here is FRR, a bgp routing software.
290 Each node in the proxmox cluster peer with each others nodes.
291 For bigger networks, or multiple proxmox clusters,
292 it's possible to use external bgp route reflector servers.
298 iface eno1 inet manual
301 iface vmbr0 inet static
303 netmask 255.255.255.0
309 iface vxlan2 inet manual
310 vxlan-local-tunnelip 192.168.0.1
312 bridge-arp-nd-suppress on
313 bridge-unicast-flood off
314 bridge-multicast-flood off
318 iface vmbr2 inet manual
325 iface vxlan3 inet manual
326 vxlan-local-tunnelip 192.168.0.1
328 bridge-arp-nd-suppress on
329 bridge-unicast-flood off
330 bridge-multicast-flood off
334 iface vmbr3 inet manual
345 no bgp default ipv4-unicast
347 neighbor 192.168.0.2 remote-as 1234
348 neighbor 192.168.0.3 remote-as 1234
350 address-family l2vpn evpn
351 neighbor 192.168.0.2 activate
352 neighbor 192.168.0.3 activate
365 iface eno1 inet manual
368 iface vmbr0 inet static
370 netmask 255.255.255.0
376 iface vxlan2 inet manual
377 vxlan-local-tunnelip 192.168.0.2
379 bridge-arp-nd-suppress on
380 bridge-unicast-flood off
381 bridge-multicast-flood off
385 iface vmbr2 inet manual
391 iface vxlan3 inet manual
392 vxlan-local-tunnelip 192.168.0.2
394 bridge-arp-nd-suppress on
395 bridge-unicast-flood off
396 bridge-multicast-flood off
400 iface vmbr3 inet manual
411 no bgp default ipv4-unicast
413 neighbor 192.168.0.1 remote-as 1234
414 neighbor 192.168.0.3 remote-as 1234
416 address-family l2vpn evpn
417 neighbor 192.168.0.1 activate
418 neighbor 192.168.0.3 activate
431 iface eno1 inet manual
434 iface vmbr0 inet static
436 netmask 255.255.255.0
442 iface vxlan2 inet manual
443 vxlan-local-tunnelip 192.168.0.3
445 bridge-arp-nd-suppress on
446 bridge-unicast-flood off
447 bridge-multicast-flood off
451 iface vmbr2 inet manual
457 iface vxlan3 inet manual
458 vxlan-local-tunnelip 192.168.0.3
460 bridge-arp-nd-suppress on
461 bridge-unicast-flood off
462 bridge-multicast-flood off
466 iface vmbr3 inet manual
478 no bgp default ipv4-unicast
480 neighbor 192.168.0.1 remote-as 1234
481 neighbor 192.168.0.2 remote-as 1234
483 address-family l2vpn evpn
484 neighbor 192.168.0.1 activate
485 neighbor 192.168.0.2 activate
493 VXLAN layer3 routing with anycast gateway
494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
496 With this need, each vmbr bridge will be the gateway for the vm.
497 Same vmbr on different node, will have same ip address and same mac address,
498 to have working vm live migration and no network disruption.
500 VXLAN layer3 routing only work with FRR and non-aware bridge.
501 (vlan aware bridge support is buggy currently).
508 net.ipv4.ip_forward=1
509 net.ipv6.conf.all.forwarding=1
510 #disable reverse path filtering
511 net.ipv4.conf.default.rp_filter=0
512 net.ipv4.conf.all.rp_filter=0
513 #allow frr to work with vrf
514 net.ipv4.tcp_l3mdev_accept=1
520 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
522 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
523 but only bridging on the egress.
524 This results in bi-directional VXLAN traffic traveling on different VNIs
525 in each direction (always the destination VNI) across the routed infrastructure.
527 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
534 iface eno1 inet manual
537 iface vmbr0 inet static
539 netmask 255.255.255.0
545 iface vxlan2 inet manual
546 vxlan-local-tunnelip 192.168.0.1
548 bridge-arp-nd-suppress on
549 bridge-unicast-flood off
550 bridge-multicast-flood off
554 iface vmbr2 inet static
556 netmask 255.255.255.0
557 hwaddress 44:39:39:FF:40:94
564 iface vxlan3 inet manual
565 vxlan-local-tunnelip 192.168.0.1
567 bridge-arp-nd-suppress on
568 bridge-unicast-flood off
569 bridge-multicast-flood off
573 iface vmbr3 inet static
575 netmask 255.255.255.0
576 hwaddress 44:39:39:FF:40:94
587 bgp router-id 192.168.0.1
588 no bgp default ipv4-unicast
590 neighbor 192.168.0.2 remote-as 1234
591 neighbor 192.168.0.3 remote-as 1234
593 address-family l2vpn evpn
594 neighbor 192.168.0.2 activate
595 neighbor 192.168.0.3 activate
608 iface eno1 inet manual
611 iface vmbr0 inet static
613 netmask 255.255.255.0
619 iface vxlan2 inet manual
620 vxlan-local-tunnelip 192.168.0.2
622 bridge-arp-nd-suppress on
623 bridge-unicast-flood off
624 bridge-multicast-flood off
628 iface vmbr2 inet static
630 netmask 255.255.255.0
631 hwaddress 44:39:39:FF:40:94
638 iface vxlan3 inet manual
639 vxlan-local-tunnelip 192.168.0.2
641 bridge-arp-nd-suppress on
642 bridge-unicast-flood off
643 bridge-multicast-flood off
647 iface vmbr3 inet static
649 netmask 255.255.255.0
650 hwaddress 44:39:39:FF:40:94
661 bgp router-id 192.168.0.2
662 no bgp default ipv4-unicast
664 neighbor 192.168.0.1 remote-as 1234
665 neighbor 192.168.0.3 remote-as 1234
667 address-family l2vpn evpn
668 neighbor 192.168.0.1 activate
669 neighbor 192.168.0.3 activate
682 iface eno1 inet manual
685 iface vmbr0 inet static
687 netmask 255.255.255.0
693 iface vxlan2 inet manual
694 vxlan-local-tunnelip 192.168.0.3
696 bridge-arp-nd-suppress on
697 bridge-unicast-flood off
698 bridge-multicast-flood off
702 iface vmbr2 inet static
704 netmask 255.255.255.0
705 hwaddress 44:39:39:FF:40:94
712 iface vxlan3 inet manual
713 vxlan-local-tunnelip 192.168.0.3
715 bridge-arp-nd-suppress on
716 bridge-unicast-flood off
717 bridge-multicast-flood off
721 iface vmbr3 inet static
723 netmask 255.255.255.0
724 hwaddress 44:39:39:FF:40:94
735 bgp router-id 192.168.0.3
736 no bgp default ipv4-unicast
738 neighbor 192.168.0.1 remote-as 1234
739 neighbor 192.168.0.2 remote-as 1234
741 address-family l2vpn evpn
742 neighbor 192.168.0.1 activate
743 neighbor 192.168.0.2 activate
755 With this model, you don't need to have all vxlan on all nodes.
756 This model will also be needed to route traffic to an external router.
758 The symmetric model routes and bridges on both the ingress and the egress leafs.
759 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
760 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
761 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
762 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
764 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
766 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
777 iface eno1 inet manual
780 iface vmbr0 inet static
782 netmask 255.255.255.0
788 iface vxlan2 inet manual
789 vxlan-local-tunnelip 192.168.0.1
791 bridge-arp-nd-suppress on
792 bridge-unicast-flood off
793 bridge-multicast-flood off
796 iface vmbr2 inet static
801 netmask 255.255.255.0
802 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
806 iface vxlan3 inet manual
807 vxlan-local-tunnelip 192.168.0.1
809 bridge-arp-nd-suppress on
810 bridge-unicast-flood off
811 bridge-multicast-flood off
814 iface vmbr3 inet static
819 netmask 255.255.255.0
820 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
823 #interconnect vxlan-vfr l3vni
825 iface vxlan4000 inet manual
826 vxlan-local-tunnelip 192.168.0.1
828 bridge-arp-nd-suppress on
829 bridge-unicast-flood off
830 bridge-multicast-flood off
834 iface vmbr4000 inet manual
835 bridge_ports vxlan4000
838 hwaddress 44:39:39:FF:40:90 #must be different on each node
849 bgp router-id 192.168.0.1
850 no bgp default ipv4-unicast
852 neighbor 192.168.0.2 remote-as 1234
853 neighbor 192.168.0.3 remote-as 1234
855 address-family l2vpn evpn
856 neighbor 192.168.0.2 activate
857 neighbor 192.168.0.3 activate
861 router bgp 1234 vrf vrf1
863 bgp router-id 192.168.0.1
865 address-family ipv4 unicast
866 redistribute connected
869 address-family l2vpn evpn
870 advertise ipv4 unicast
886 iface eno1 inet manual
889 iface vmbr0 inet static
891 netmask 255.255.255.0
897 iface vxlan2 inet manual
898 vxlan-local-tunnelip 192.168.0.2
900 bridge-arp-nd-suppress on
901 bridge-unicast-flood off
902 bridge-multicast-flood off
905 iface vmbr2 inet static
910 netmask 255.255.255.0
911 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
915 iface vxlan3 inet manual
916 vxlan-local-tunnelip 192.168.0.2
918 bridge-arp-nd-suppress on
919 bridge-unicast-flood off
920 bridge-multicast-flood off
923 iface vmbr3 inet static
928 netmask 255.255.255.0
929 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
932 #interconnect vxlan-vfr l3vni
934 iface vxlan4000 inet manual
935 vxlan-local-tunnelip 192.168.0.2
937 bridge-arp-nd-suppress on
938 bridge-unicast-flood off
939 bridge-multicast-flood off
943 iface vmbr4000 inet manual
944 bridge_ports vxlan4000
947 hwaddress 44:39:39:FF:40:91 #must be different on each node
959 bgp router-id 192.168.0.2
960 no bgp default ipv4-unicast
962 neighbor 192.168.0.1 remote-as 1234
963 neighbor 192.168.0.3 remote-as 1234
965 address-family l2vpn evpn
966 neighbor 192.168.0.1 activate
967 neighbor 192.168.0.3 activate
971 router bgp 1234 vrf vrf1
973 bgp router-id 192.168.0.2
975 address-family ipv4 unicast
976 redistribute connected
979 address-family l2vpn evpn
980 advertise ipv4 unicast
996 iface eno1 inet manual
999 iface vmbr0 inet static
1001 netmask 255.255.255.0
1007 iface vxlan2 inet manual
1008 vxlan-local-tunnelip 192.168.0.3
1010 bridge-arp-nd-suppress on
1011 bridge-unicast-flood off
1012 bridge-multicast-flood off
1015 iface vmbr2 inet static
1020 netmask 255.255.255.0
1021 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1025 iface vxlan3 inet manual
1026 vxlan-local-tunnelip 192.168.0.3
1028 bridge-arp-nd-suppress on
1029 bridge-unicast-flood off
1030 bridge-multicast-flood off
1033 iface vmbr3 inet static
1038 netmask 255.255.255.0
1039 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1042 #interconnect vxlan-vfr l3vni
1044 iface vxlan4000 inet manual
1045 vxlan-local-tunnelip 192.168.0.3
1047 bridge-arp-nd-suppress on
1048 bridge-unicast-flood off
1049 bridge-multicast-flood off
1053 iface vmbr4000 inet manual
1054 bridge_ports vxlan4000
1057 hwaddress 44:39:39:FF:40:92 #must be different on each node
1069 bgp router-id 192.168.0.3
1070 no bgp default ipv4-unicast
1072 neighbor 192.168.0.1 remote-as 1234
1073 neighbor 192.168.0.2 remote-as 1234
1075 address-family l2vpn evpn
1076 neighbor 192.168.0.1 activate
1077 neighbor 192.168.0.2 activate
1081 router bgp 1234 vrf vrf1
1083 bgp router-id 192.168.0.3
1085 address-family ipv4 unicast
1086 redistribute connected
1089 address-family l2vpn evpn
1090 advertise ipv4 unicast