4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
27 This scenario relies in head end replication, meaning that end host in case
28 of not having any entry for the destination MAC address will send out an ARP
29 to other devices / VTEPs in the VXLAN network.
30 This is done by sending the request to the VXLAN multicast group,
31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
38 iface eno1 inet manual
41 iface vmbr0 inet static
49 iface vxlan2 inet manual
50 vxlan-svcnodeip 225.20.1.1
54 iface vmbr2 inet manual
60 iface vxlan3 inet manual
61 vxlan-svcnodeip 225.20.1.1
65 iface vmbr3 inet manual
76 iface eno1 inet manual
79 iface vmbr0 inet static
87 iface vxlan2 inet manual
88 vxlan-svcnodeip 225.20.1.1
92 iface vmbr2 inet manual
99 iface vxlan3 inet manual
100 vxlan-svcnodeip 225.20.1.1
104 iface vmbr3 inet manual
115 iface eno1 inet manual
118 iface vmbr0 inet static
120 netmask 255.255.255.0
126 iface vxlan2 inet manual
127 vxlan-svcnodeip 225.20.1.1
131 iface vmbr2 inet manual
138 iface vxlan3 inet manual
139 vxlan-svcnodeip 225.20.1.1
143 iface vmbr3 inet manual
153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
154 The VXLAN is defined without a remote multicast group.
155 Instead, all the remote VTEPs are associated with the all-zero address:
156 a BUM frame will be duplicated to all these destinations.
157 The VXLAN device will still learn remote addresses automatically using source-address learning.
163 iface eno1 inet manual
166 iface vmbr0 inet static
168 netmask 255.255.255.0
175 iface vxlan2 inet manual
176 vxlan_remoteip 192.168.0.2
177 vxlan_remoteip 192.168.0.3
181 iface vmbr2 inet manual
188 iface vxlan2 inet manual
189 vxlan_remoteip 192.168.0.2
190 vxlan_remoteip 192.168.0.3
194 iface vmbr3 inet manual
205 iface eno1 inet manual
208 iface vmbr0 inet static
210 netmask 255.255.255.0
216 iface vxlan2 inet manual
217 vxlan_remoteip 192.168.0.1
218 vxlan_remoteip 192.168.0.3
223 iface vmbr2 inet manual
229 iface vxlan2 inet manual
230 vxlan_remoteip 192.168.0.1
231 vxlan_remoteip 192.168.0.3
235 iface vmbr3 inet manual
246 iface eno1 inet manual
249 iface vmbr0 inet static
251 netmask 255.255.255.0
257 iface vxlan2 inet manual
258 vxlan_remoteip 192.168.0.2
259 vxlan_remoteip 192.168.0.3
264 iface vmbr2 inet manual
270 iface vxlan2 inet manual
271 vxlan_remoteip 192.168.0.2
272 vxlan_remoteip 192.168.0.3
276 iface vmbr3 inet manual
286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
289 The control plane used here is FRR, a bgp routing software.
290 Each node in the proxmox cluster peer with each others nodes.
291 For bigger networks, or multiple proxmox clusters,
292 it's possible to use external bgp route reflector servers.
298 iface eno1 inet manual
301 iface vmbr0 inet static
303 netmask 255.255.255.0
309 iface vxlan2 inet manual
310 vxlan-local-tunnelip 192.168.0.1
312 bridge-arp-nd-suppress on
313 bridge-unicast-flood off
314 bridge-multicast-flood off
318 iface vmbr2 inet manual
325 iface vxlan3 inet manual
326 vxlan-local-tunnelip 192.168.0.1
328 bridge-arp-nd-suppress on
329 bridge-unicast-flood off
330 bridge-multicast-flood off
334 iface vmbr3 inet manual
345 no bgp default ipv4-unicast
347 neighbor 192.168.0.2 remote-as 1234
348 neighbor 192.168.0.3 remote-as 1234
350 address-family l2vpn evpn
351 neighbor 192.168.0.2 activate
352 neighbor 192.168.0.3 activate
365 iface eno1 inet manual
368 iface vmbr0 inet static
370 netmask 255.255.255.0
376 iface vxlan2 inet manual
377 vxlan-local-tunnelip 192.168.0.2
379 bridge-arp-nd-suppress on
380 bridge-unicast-flood off
381 bridge-multicast-flood off
385 iface vmbr2 inet manual
391 iface vxlan3 inet manual
392 vxlan-local-tunnelip 192.168.0.2
394 bridge-arp-nd-suppress on
395 bridge-unicast-flood off
396 bridge-multicast-flood off
400 iface vmbr3 inet manual
411 no bgp default ipv4-unicast
413 neighbor 192.168.0.1 remote-as 1234
414 neighbor 192.168.0.3 remote-as 1234
416 address-family l2vpn evpn
417 neighbor 192.168.0.1 activate
418 neighbor 192.168.0.3 activate
431 iface eno1 inet manual
434 iface vmbr0 inet static
436 netmask 255.255.255.0
442 iface vxlan2 inet manual
443 vxlan-local-tunnelip 192.168.0.3
445 bridge-arp-nd-suppress on
446 bridge-unicast-flood off
447 bridge-multicast-flood off
451 iface vmbr2 inet manual
457 iface vxlan3 inet manual
458 vxlan-local-tunnelip 192.168.0.3
460 bridge-arp-nd-suppress on
461 bridge-unicast-flood off
462 bridge-multicast-flood off
466 iface vmbr3 inet manual
478 no bgp default ipv4-unicast
480 neighbor 192.168.0.1 remote-as 1234
481 neighbor 192.168.0.2 remote-as 1234
483 address-family l2vpn evpn
484 neighbor 192.168.0.1 activate
485 neighbor 192.168.0.2 activate
493 VXLAN layer3 routing with anycast gateway
494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
496 With this need, each vmbr bridge will be the gateway for the vm.
497 Same vmbr on different node, will have same ip address and same mac address,
498 to have working vm live migration and no network disruption.
500 VXLAN layer3 routing only work with FRR and non-aware bridge.
501 (vlan aware bridge support is buggy currently).
506 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
508 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
509 but only bridging on the egress.
510 This results in bi-directional VXLAN traffic traveling on different VNIs
511 in each direction (always the destination VNI) across the routed infrastructure.
513 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
520 net.ipv4.ip_forward=1
521 net.ipv6.conf.all.forwarding=1
528 iface eno1 inet manual
531 iface vmbr0 inet static
533 netmask 255.255.255.0
539 iface vxlan2 inet manual
540 vxlan-local-tunnelip 192.168.0.1
542 bridge-arp-nd-suppress on
543 bridge-unicast-flood off
544 bridge-multicast-flood off
548 iface vmbr2 inet static
550 netmask 255.255.255.0
551 hwaddress 44:39:39:FF:40:94
558 iface vxlan3 inet manual
559 vxlan-local-tunnelip 192.168.0.1
561 bridge-arp-nd-suppress on
562 bridge-unicast-flood off
563 bridge-multicast-flood off
567 iface vmbr3 inet static
569 netmask 255.255.255.0
570 hwaddress 44:39:39:FF:40:94
581 bgp router-id 192.168.0.1
582 no bgp default ipv4-unicast
584 neighbor 192.168.0.2 remote-as 1234
585 neighbor 192.168.0.3 remote-as 1234
587 address-family l2vpn evpn
588 neighbor 192.168.0.2 activate
589 neighbor 192.168.0.3 activate
602 iface eno1 inet manual
605 iface vmbr0 inet static
607 netmask 255.255.255.0
613 iface vxlan2 inet manual
614 vxlan-local-tunnelip 192.168.0.2
616 bridge-arp-nd-suppress on
617 bridge-unicast-flood off
618 bridge-multicast-flood off
622 iface vmbr2 inet static
624 netmask 255.255.255.0
625 hwaddress 44:39:39:FF:40:94
632 iface vxlan3 inet manual
633 vxlan-local-tunnelip 192.168.0.2
635 bridge-arp-nd-suppress on
636 bridge-unicast-flood off
637 bridge-multicast-flood off
641 iface vmbr3 inet static
643 netmask 255.255.255.0
644 hwaddress 44:39:39:FF:40:94
655 bgp router-id 192.168.0.2
656 no bgp default ipv4-unicast
658 neighbor 192.168.0.1 remote-as 1234
659 neighbor 192.168.0.3 remote-as 1234
661 address-family l2vpn evpn
662 neighbor 192.168.0.1 activate
663 neighbor 192.168.0.3 activate
676 iface eno1 inet manual
679 iface vmbr0 inet static
681 netmask 255.255.255.0
687 iface vxlan2 inet manual
688 vxlan-local-tunnelip 192.168.0.3
690 bridge-arp-nd-suppress on
691 bridge-unicast-flood off
692 bridge-multicast-flood off
696 iface vmbr2 inet static
698 netmask 255.255.255.0
699 hwaddress 44:39:39:FF:40:94
706 iface vxlan3 inet manual
707 vxlan-local-tunnelip 192.168.0.3
709 bridge-arp-nd-suppress on
710 bridge-unicast-flood off
711 bridge-multicast-flood off
715 iface vmbr3 inet static
717 netmask 255.255.255.0
718 hwaddress 44:39:39:FF:40:94
729 bgp router-id 192.168.0.3
730 no bgp default ipv4-unicast
732 neighbor 192.168.0.1 remote-as 1234
733 neighbor 192.168.0.2 remote-as 1234
735 address-family l2vpn evpn
736 neighbor 192.168.0.1 activate
737 neighbor 192.168.0.2 activate
749 With this model, you don't need to have all vxlan on all nodes.
750 This model will also be needed to route traffic to an external router.
752 The symmetric model routes and bridges on both the ingress and the egress leafs.
753 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
754 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
755 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
756 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
758 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
760 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
766 net.ipv4.ip_forward=1
767 net.ipv6.conf.all.forwarding=1
768 #disable reverse path filtering
769 net.ipv4.conf.default.rp_filter=0
770 net.ipv4.conf.all.rp_filter=0
781 iface eno1 inet manual
784 iface vmbr0 inet static
786 netmask 255.255.255.0
792 iface vxlan2 inet manual
793 vxlan-local-tunnelip 192.168.0.1
795 bridge-arp-nd-suppress on
796 bridge-unicast-flood off
797 bridge-multicast-flood off
800 iface vmbr2 inet static
805 netmask 255.255.255.0
806 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
810 iface vxlan3 inet manual
811 vxlan-local-tunnelip 192.168.0.1
813 bridge-arp-nd-suppress on
814 bridge-unicast-flood off
815 bridge-multicast-flood off
818 iface vmbr3 inet static
823 netmask 255.255.255.0
824 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
827 #interconnect vxlan-vfr l3vni
829 iface vxlan4000 inet manual
830 vxlan-local-tunnelip 192.168.0.1
832 bridge-arp-nd-suppress on
833 bridge-unicast-flood off
834 bridge-multicast-flood off
838 iface vmbr4000 inet manual
839 bridge_ports vxlan4000
842 hwaddress 44:39:39:FF:40:90 #must be different on each node
853 bgp router-id 192.168.0.1
854 no bgp default ipv4-unicast
856 neighbor 192.168.0.2 remote-as 1234
857 neighbor 192.168.0.3 remote-as 1234
859 address-family l2vpn evpn
860 neighbor 192.168.0.2 activate
861 neighbor 192.168.0.3 activate
865 router bgp 1234 vrf vrf1
867 bgp router-id 192.168.0.1
869 address-family ipv4 unicast
870 redistribute connected
873 address-family l2vpn evpn
874 advertise ipv4 unicast
890 iface eno1 inet manual
893 iface vmbr0 inet static
895 netmask 255.255.255.0
901 iface vxlan2 inet manual
902 vxlan-local-tunnelip 192.168.0.2
904 bridge-arp-nd-suppress on
905 bridge-unicast-flood off
906 bridge-multicast-flood off
909 iface vmbr2 inet static
914 netmask 255.255.255.0
915 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
919 iface vxlan3 inet manual
920 vxlan-local-tunnelip 192.168.0.2
922 bridge-arp-nd-suppress on
923 bridge-unicast-flood off
924 bridge-multicast-flood off
927 iface vmbr3 inet static
932 netmask 255.255.255.0
933 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
936 #interconnect vxlan-vfr l3vni
938 iface vxlan4000 inet manual
939 vxlan-local-tunnelip 192.168.0.2
941 bridge-arp-nd-suppress on
942 bridge-unicast-flood off
943 bridge-multicast-flood off
947 iface vmbr4000 inet manual
948 bridge_ports vxlan4000
951 hwaddress 44:39:39:FF:40:91 #must be different on each node
963 bgp router-id 192.168.0.2
964 no bgp default ipv4-unicast
966 neighbor 192.168.0.1 remote-as 1234
967 neighbor 192.168.0.3 remote-as 1234
969 address-family l2vpn evpn
970 neighbor 192.168.0.1 activate
971 neighbor 192.168.0.3 activate
975 router bgp 1234 vrf vrf1
977 bgp router-id 192.168.0.2
979 address-family ipv4 unicast
980 redistribute connected
983 address-family l2vpn evpn
984 advertise ipv4 unicast
1000 iface eno1 inet manual
1003 iface vmbr0 inet static
1005 netmask 255.255.255.0
1011 iface vxlan2 inet manual
1012 vxlan-local-tunnelip 192.168.0.3
1014 bridge-arp-nd-suppress on
1015 bridge-unicast-flood off
1016 bridge-multicast-flood off
1019 iface vmbr2 inet static
1024 netmask 255.255.255.0
1025 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1029 iface vxlan3 inet manual
1030 vxlan-local-tunnelip 192.168.0.3
1032 bridge-arp-nd-suppress on
1033 bridge-unicast-flood off
1034 bridge-multicast-flood off
1037 iface vmbr3 inet static
1042 netmask 255.255.255.0
1043 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1046 #interconnect vxlan-vfr l3vni
1048 iface vxlan4000 inet manual
1049 vxlan-local-tunnelip 192.168.0.3
1051 bridge-arp-nd-suppress on
1052 bridge-unicast-flood off
1053 bridge-multicast-flood off
1057 iface vmbr4000 inet manual
1058 bridge_ports vxlan4000
1061 hwaddress 44:39:39:FF:40:92 #must be different on each node
1073 bgp router-id 192.168.0.3
1074 no bgp default ipv4-unicast
1076 neighbor 192.168.0.1 remote-as 1234
1077 neighbor 192.168.0.2 remote-as 1234
1079 address-family l2vpn evpn
1080 neighbor 192.168.0.1 activate
1081 neighbor 192.168.0.2 activate
1085 router bgp 1234 vrf vrf1
1087 bgp router-id 192.168.0.3
1089 address-family ipv4 unicast
1090 redistribute connected
1093 address-family l2vpn evpn
1094 advertise ipv4 unicast
1101 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1103 Routing to outside need the symmetric model.
1107 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1108 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1109 and announce this default gw to other proxmox nodes.
1120 iface eno1 inet manual
1123 iface vmbr0 inet static
1125 netmask 255.255.255.0
1133 netmask 255.255.255.0
1135 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1136 #if you have multiple external routers, you can use ecmp balancing
1137 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1140 iface vxlan2 inet manual
1141 vxlan-local-tunnelip 192.168.0.1
1143 bridge-arp-nd-suppress on
1144 bridge-unicast-flood off
1145 bridge-multicast-flood off
1148 iface vmbr2 inet static
1153 netmask 255.255.255.0
1154 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1158 iface vxlan3 inet manual
1159 vxlan-local-tunnelip 192.168.0.1
1161 bridge-arp-nd-suppress on
1162 bridge-unicast-flood off
1163 bridge-multicast-flood off
1166 iface vmbr3 inet static
1171 netmask 255.255.255.0
1172 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1175 #interconnect vxlan-vfr l3vni
1177 iface vxlan4000 inet manual
1178 vxlan-local-tunnelip 192.168.0.1
1180 bridge-arp-nd-suppress on
1181 bridge-unicast-flood off
1182 bridge-multicast-flood off
1185 iface vmbr4000 inet manual
1186 bridge_ports vxlan4000
1189 hwaddress 44:39:39:FF:40:90 #must be different on each node
1201 bgp router-id 192.168.0.1
1202 no bgp default ipv4-unicast
1204 neighbor 192.168.0.2 remote-as 1234
1205 neighbor 192.168.0.3 remote-as 1234
1207 address-family l2vpn evpn
1208 neighbor 192.168.0.2 activate
1209 neighbor 192.168.0.3 activate
1213 router bgp 1234 vrf vrf1
1215 bgp router-id 172.16.0.1
1217 address-family ipv4 unicast
1218 redistribute connected
1219 redistribute kernel !announce your default gw to all nodes
1222 address-family l2vpn evpn
1223 advertise ipv4 unicast
1239 iface eno1 inet manual
1242 iface vmbr0 inet static
1244 netmask 255.255.255.0
1250 iface vxlan2 inet manual
1251 vxlan-local-tunnelip 192.168.0.2
1253 bridge-arp-nd-suppress on
1254 bridge-unicast-flood off
1255 bridge-multicast-flood off
1258 iface vmbr2 inet static
1263 netmask 255.255.255.0
1264 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1268 iface vxlan3 inet manual
1269 vxlan-local-tunnelip 192.168.0.2
1271 bridge-arp-nd-suppress on
1272 bridge-unicast-flood off
1273 bridge-multicast-flood off
1276 iface vmbr3 inet static
1281 netmask 255.255.255.0
1282 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1285 #interconnect vxlan-vfr l3vni
1287 iface vxlan4000 inet manual
1288 vxlan-local-tunnelip 192.168.0.2
1290 bridge-arp-nd-suppress on
1291 bridge-unicast-flood off
1292 bridge-multicast-flood off
1296 iface vmbr4000 inet manual
1297 bridge_ports vxlan4000
1300 hwaddress 44:39:39:FF:40:91 #must be different on each node
1312 bgp router-id 192.168.0.2
1313 no bgp default ipv4-unicast
1315 neighbor 192.168.0.1 remote-as 1234
1316 neighbor 192.168.0.3 remote-as 1234
1318 address-family l2vpn evpn
1319 neighbor 192.168.0.1 activate
1320 neighbor 192.168.0.3 activate
1324 router bgp 1234 vrf vrf1
1326 bgp router-id 192.168.0.2
1328 address-family ipv4 unicast
1329 redistribute connected
1332 address-family l2vpn evpn
1333 advertise ipv4 unicast
1349 iface eno1 inet manual
1352 iface vmbr0 inet static
1354 netmask 255.255.255.0
1360 iface vxlan2 inet manual
1361 vxlan-local-tunnelip 192.168.0.3
1363 bridge-arp-nd-suppress on
1364 bridge-unicast-flood off
1365 bridge-multicast-flood off
1368 iface vmbr2 inet static
1373 netmask 255.255.255.0
1374 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1378 iface vxlan3 inet manual
1379 vxlan-local-tunnelip 192.168.0.3
1381 bridge-arp-nd-suppress on
1382 bridge-unicast-flood off
1383 bridge-multicast-flood off
1386 iface vmbr3 inet static
1391 netmask 255.255.255.0
1392 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1395 #interconnect vxlan-vfr l3vni
1397 iface vxlan4000 inet manual
1398 vxlan-local-tunnelip 192.168.0.3
1400 bridge-arp-nd-suppress on
1401 bridge-unicast-flood off
1402 bridge-multicast-flood off
1406 iface vmbr4000 inet manual
1407 bridge_ports vxlan4000
1410 hwaddress 44:39:39:FF:40:92 #must be different on each node
1422 bgp router-id 192.168.0.3
1423 no bgp default ipv4-unicast
1425 neighbor 192.168.0.1 remote-as 1234
1426 neighbor 192.168.0.2 remote-as 1234
1428 address-family l2vpn evpn
1429 neighbor 192.168.0.1 activate
1430 neighbor 192.168.0.2 activate
1434 router bgp 1234 vrf vrf1
1436 bgp router-id 192.168.0.3
1438 address-family ipv4 unicast
1439 redistribute connected
1442 address-family l2vpn evpn
1443 advertise ipv4 unicast
1450 multiple gateway nodes
1451 ^^^^^^^^^^^^^^^^^^^^^^
1452 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1453 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1454 and announce this default gw.
1455 The external router have ecmp routes to all proxmox nodes.(balancing).
1456 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1457 vxlan the packet to final destination.
1467 iface eno1 inet manual
1470 iface vmbr0 inet static
1472 netmask 255.255.255.0
1480 netmask 255.255.255.0
1482 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1483 #if you have multiple external routers, you can use ecmp balancing
1484 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1487 iface vxlan2 inet manual
1488 vxlan-local-tunnelip 192.168.0.1
1490 bridge-arp-nd-suppress on
1491 bridge-unicast-flood off
1492 bridge-multicast-flood off
1495 iface vmbr2 inet static
1500 netmask 255.255.255.0
1501 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1505 iface vxlan3 inet manual
1506 vxlan-local-tunnelip 192.168.0.1
1508 bridge-arp-nd-suppress on
1509 bridge-unicast-flood off
1510 bridge-multicast-flood off
1513 iface vmbr3 inet static
1518 netmask 255.255.255.0
1519 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1522 #interconnect vxlan-vfr l3vni
1524 iface vxlan4000 inet manual
1525 vxlan-local-tunnelip 192.168.0.1
1527 bridge-arp-nd-suppress on
1528 bridge-unicast-flood off
1529 bridge-multicast-flood off
1532 iface vmbr4000 inet manual
1533 bridge_ports vxlan4000
1536 hwaddress 44:39:39:FF:40:90 #must be different on each node
1548 bgp router-id 192.168.0.1
1549 no bgp default ipv4-unicast
1551 neighbor 192.168.0.2 remote-as 1234
1552 neighbor 192.168.0.3 remote-as 1234
1554 address-family l2vpn evpn
1555 neighbor 192.168.0.2 activate
1556 neighbor 192.168.0.3 activate
1560 router bgp 1234 vrf vrf1
1562 bgp router-id 172.16.0.1
1564 address-family ipv4 unicast
1565 redistribute connected
1566 redistribute kernel !announce your default gw to all nodes
1569 address-family l2vpn evpn
1570 advertise ipv4 unicast
1586 iface eno1 inet manual
1589 iface vmbr0 inet static
1591 netmask 255.255.255.0
1599 netmask 255.255.255.0
1601 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1602 #if you have multiple external routers, you can use ecmp balancing
1603 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1606 iface vxlan2 inet manual
1607 vxlan-local-tunnelip 192.168.0.2
1609 bridge-arp-nd-suppress on
1610 bridge-unicast-flood off
1611 bridge-multicast-flood off
1614 iface vmbr2 inet static
1619 netmask 255.255.255.0
1620 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1624 iface vxlan3 inet manual
1625 vxlan-local-tunnelip 192.168.0.2
1627 bridge-arp-nd-suppress on
1628 bridge-unicast-flood off
1629 bridge-multicast-flood off
1632 iface vmbr3 inet static
1637 netmask 255.255.255.0
1638 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1641 #interconnect vxlan-vfr l3vni
1643 iface vxlan4000 inet manual
1644 vxlan-local-tunnelip 192.168.0.2
1646 bridge-arp-nd-suppress on
1647 bridge-unicast-flood off
1648 bridge-multicast-flood off
1652 iface vmbr4000 inet manual
1653 bridge_ports vxlan4000
1656 hwaddress 44:39:39:FF:40:91 #must be different on each node
1668 bgp router-id 192.168.0.2
1669 no bgp default ipv4-unicast
1671 neighbor 192.168.0.1 remote-as 1234
1672 neighbor 192.168.0.3 remote-as 1234
1674 address-family l2vpn evpn
1675 neighbor 192.168.0.1 activate
1676 neighbor 192.168.0.3 activate
1680 router bgp 1234 vrf vrf1
1682 bgp router-id 172.16.0.2
1684 address-family ipv4 unicast
1685 redistribute connected
1686 redistribute kernel !announce your default gw to all nodes
1689 address-family l2vpn evpn
1690 advertise ipv4 unicast
1706 iface eno1 inet manual
1709 iface vmbr0 inet static
1711 netmask 255.255.255.0
1719 netmask 255.255.255.0
1721 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1722 #if you have multiple external routers, you can use ecmp balancing
1723 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1726 iface vxlan2 inet manual
1727 vxlan-local-tunnelip 192.168.0.3
1729 bridge-arp-nd-suppress on
1730 bridge-unicast-flood off
1731 bridge-multicast-flood off
1734 iface vmbr2 inet static
1739 netmask 255.255.255.0
1740 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1744 iface vxlan3 inet manual
1745 vxlan-local-tunnelip 192.168.0.3
1747 bridge-arp-nd-suppress on
1748 bridge-unicast-flood off
1749 bridge-multicast-flood off
1752 iface vmbr3 inet static
1757 netmask 255.255.255.0
1758 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1761 #interconnect vxlan-vfr l3vni
1763 iface vxlan4000 inet manual
1764 vxlan-local-tunnelip 192.168.0.3
1766 bridge-arp-nd-suppress on
1767 bridge-unicast-flood off
1768 bridge-multicast-flood off
1772 iface vmbr4000 inet manual
1773 bridge_ports vxlan4000
1776 hwaddress 44:39:39:FF:40:92 #must be different on each node
1788 bgp router-id 192.168.0.3
1789 no bgp default ipv4-unicast
1791 neighbor 192.168.0.1 remote-as 1234
1792 neighbor 192.168.0.2 remote-as 1234
1794 address-family l2vpn evpn
1795 neighbor 192.168.0.1 activate
1796 neighbor 192.168.0.2 activate
1800 router bgp 1234 vrf vrf1
1802 bgp router-id 172.16.0.3
1804 address-family ipv4 unicast
1805 redistribute connected
1806 redistribute kernel !announce your default gw to all nodes
1809 address-family l2vpn evpn
1810 advertise ipv4 unicast
1820 If your external router don't support ecmp to reach multiple proxmox nodes,
1821 you can setup an HA floating vip on proxmox nodes with vrrp
1823 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1824 Node1 is the primary and failover to node2 in case of failure.
1833 netmask 255.255.255.0
1835 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1838 vrrp-virtual-ip 172.16.0.10
1847 netmask 255.255.255.0
1849 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1852 vrrp-virtual-ip 172.16.0.10