4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
30 This scenario relies in head end replication, meaning that end host in case
31 of not having any entry for the destination MAC address will send out an ARP
32 to other devices / VTEPs in the VXLAN network.
33 This is done by sending the request to the VXLAN multicast group,
34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
41 iface eno1 inet manual
45 iface vmbr0 inet static
53 iface vxlan2 inet manual
54 vxlan-svcnodeip 225.20.1.1
58 iface vmbr2 inet manual
64 iface vxlan3 inet manual
65 vxlan-svcnodeip 225.20.1.1
69 iface vmbr3 inet manual
80 iface eno1 inet manual
84 iface vmbr0 inet static
92 iface vxlan2 inet manual
93 vxlan-svcnodeip 225.20.1.1
97 iface vmbr2 inet manual
104 iface vxlan3 inet manual
105 vxlan-svcnodeip 225.20.1.1
109 iface vmbr3 inet manual
120 iface eno1 inet manual
124 iface vmbr0 inet static
126 netmask 255.255.255.0
132 iface vxlan2 inet manual
133 vxlan-svcnodeip 225.20.1.1
137 iface vmbr2 inet manual
144 iface vxlan3 inet manual
145 vxlan-svcnodeip 225.20.1.1
149 iface vmbr3 inet manual
159 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
160 The VXLAN is defined without a remote multicast group.
161 Instead, all the remote VTEPs are associated with the all-zero address:
162 a BUM frame will be duplicated to all these destinations.
163 The VXLAN device will still learn remote addresses automatically using source-address learning.
169 iface eno1 inet manual
173 iface vmbr0 inet static
175 netmask 255.255.255.0
182 iface vxlan2 inet manual
183 vxlan_remoteip 192.168.0.2
184 vxlan_remoteip 192.168.0.3
188 iface vmbr2 inet manual
195 iface vxlan2 inet manual
196 vxlan_remoteip 192.168.0.2
197 vxlan_remoteip 192.168.0.3
201 iface vmbr3 inet manual
212 iface eno1 inet manual
216 iface vmbr0 inet static
218 netmask 255.255.255.0
224 iface vxlan2 inet manual
225 vxlan_remoteip 192.168.0.1
226 vxlan_remoteip 192.168.0.3
231 iface vmbr2 inet manual
237 iface vxlan2 inet manual
238 vxlan_remoteip 192.168.0.1
239 vxlan_remoteip 192.168.0.3
243 iface vmbr3 inet manual
254 iface eno1 inet manual
258 iface vmbr0 inet static
260 netmask 255.255.255.0
266 iface vxlan2 inet manual
267 vxlan_remoteip 192.168.0.2
268 vxlan_remoteip 192.168.0.3
273 iface vmbr2 inet manual
279 iface vxlan2 inet manual
280 vxlan_remoteip 192.168.0.2
281 vxlan_remoteip 192.168.0.3
285 iface vmbr3 inet manual
295 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
296 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
298 The control plane used here is FRR, a bgp routing software.
299 Each node in the proxmox cluster peer with each others nodes.
300 For bigger networks, or multiple proxmox clusters,
301 it's possible to use external bgp route reflector servers.
307 iface eno1 inet manual
311 iface vmbr0 inet static
313 netmask 255.255.255.0
319 iface vxlan2 inet manual
320 vxlan-local-tunnelip 192.168.0.1
322 bridge-arp-nd-suppress on
323 bridge-unicast-flood off
324 bridge-multicast-flood off
328 iface vmbr2 inet manual
335 iface vxlan3 inet manual
336 vxlan-local-tunnelip 192.168.0.1
338 bridge-arp-nd-suppress on
339 bridge-unicast-flood off
340 bridge-multicast-flood off
344 iface vmbr3 inet manual
355 no bgp default ipv4-unicast
357 neighbor 192.168.0.2 remote-as 1234
358 neighbor 192.168.0.3 remote-as 1234
360 address-family l2vpn evpn
361 neighbor 192.168.0.2 activate
362 neighbor 192.168.0.3 activate
375 iface eno1 inet manual
379 iface vmbr0 inet static
381 netmask 255.255.255.0
387 iface vxlan2 inet manual
388 vxlan-local-tunnelip 192.168.0.2
390 bridge-arp-nd-suppress on
391 bridge-unicast-flood off
392 bridge-multicast-flood off
396 iface vmbr2 inet manual
402 iface vxlan3 inet manual
403 vxlan-local-tunnelip 192.168.0.2
405 bridge-arp-nd-suppress on
406 bridge-unicast-flood off
407 bridge-multicast-flood off
411 iface vmbr3 inet manual
422 no bgp default ipv4-unicast
424 neighbor 192.168.0.1 remote-as 1234
425 neighbor 192.168.0.3 remote-as 1234
427 address-family l2vpn evpn
428 neighbor 192.168.0.1 activate
429 neighbor 192.168.0.3 activate
442 iface eno1 inet manual
446 iface vmbr0 inet static
448 netmask 255.255.255.0
454 iface vxlan2 inet manual
455 vxlan-local-tunnelip 192.168.0.3
457 bridge-arp-nd-suppress on
458 bridge-unicast-flood off
459 bridge-multicast-flood off
463 iface vmbr2 inet manual
469 iface vxlan3 inet manual
470 vxlan-local-tunnelip 192.168.0.3
472 bridge-arp-nd-suppress on
473 bridge-unicast-flood off
474 bridge-multicast-flood off
478 iface vmbr3 inet manual
490 no bgp default ipv4-unicast
492 neighbor 192.168.0.1 remote-as 1234
493 neighbor 192.168.0.2 remote-as 1234
495 address-family l2vpn evpn
496 neighbor 192.168.0.1 activate
497 neighbor 192.168.0.2 activate
505 VXLAN layer3 routing with anycast gateway
506 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
508 With this need, each vmbr bridge will be the gateway for the vm.
509 Same vmbr on different node, will have same ip address and same mac address,
510 to have working vm live migration and no network disruption.
512 VXLAN layer3 routing only work with FRR and non-aware bridge.
513 (vlan aware bridge support is buggy currently).
518 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
520 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
521 but only bridging on the egress.
522 This results in bi-directional VXLAN traffic traveling on different VNIs
523 in each direction (always the destination VNI) across the routed infrastructure.
525 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
532 net.ipv4.ip_forward=1
533 net.ipv6.conf.all.forwarding=1
540 iface eno1 inet manual
544 iface vmbr0 inet static
546 netmask 255.255.255.0
552 iface vxlan2 inet manual
553 vxlan-local-tunnelip 192.168.0.1
555 bridge-arp-nd-suppress on
556 bridge-unicast-flood off
557 bridge-multicast-flood off
561 iface vmbr2 inet static
563 netmask 255.255.255.0
564 hwaddress 44:39:39:FF:40:94
571 iface vxlan3 inet manual
572 vxlan-local-tunnelip 192.168.0.1
574 bridge-arp-nd-suppress on
575 bridge-unicast-flood off
576 bridge-multicast-flood off
580 iface vmbr3 inet static
582 netmask 255.255.255.0
583 hwaddress 44:39:39:FF:40:94
594 bgp router-id 192.168.0.1
595 no bgp default ipv4-unicast
597 neighbor 192.168.0.2 remote-as 1234
598 neighbor 192.168.0.3 remote-as 1234
600 address-family l2vpn evpn
601 neighbor 192.168.0.2 activate
602 neighbor 192.168.0.3 activate
615 iface eno1 inet manual
619 iface vmbr0 inet static
621 netmask 255.255.255.0
627 iface vxlan2 inet manual
628 vxlan-local-tunnelip 192.168.0.2
630 bridge-arp-nd-suppress on
631 bridge-unicast-flood off
632 bridge-multicast-flood off
636 iface vmbr2 inet static
638 netmask 255.255.255.0
639 hwaddress 44:39:39:FF:40:94
646 iface vxlan3 inet manual
647 vxlan-local-tunnelip 192.168.0.2
649 bridge-arp-nd-suppress on
650 bridge-unicast-flood off
651 bridge-multicast-flood off
655 iface vmbr3 inet static
657 netmask 255.255.255.0
658 hwaddress 44:39:39:FF:40:94
669 bgp router-id 192.168.0.2
670 no bgp default ipv4-unicast
672 neighbor 192.168.0.1 remote-as 1234
673 neighbor 192.168.0.3 remote-as 1234
675 address-family l2vpn evpn
676 neighbor 192.168.0.1 activate
677 neighbor 192.168.0.3 activate
690 iface eno1 inet manual
694 iface vmbr0 inet static
696 netmask 255.255.255.0
702 iface vxlan2 inet manual
703 vxlan-local-tunnelip 192.168.0.3
705 bridge-arp-nd-suppress on
706 bridge-unicast-flood off
707 bridge-multicast-flood off
711 iface vmbr2 inet static
713 netmask 255.255.255.0
714 hwaddress 44:39:39:FF:40:94
721 iface vxlan3 inet manual
722 vxlan-local-tunnelip 192.168.0.3
724 bridge-arp-nd-suppress on
725 bridge-unicast-flood off
726 bridge-multicast-flood off
730 iface vmbr3 inet static
732 netmask 255.255.255.0
733 hwaddress 44:39:39:FF:40:94
744 bgp router-id 192.168.0.3
745 no bgp default ipv4-unicast
747 neighbor 192.168.0.1 remote-as 1234
748 neighbor 192.168.0.2 remote-as 1234
750 address-family l2vpn evpn
751 neighbor 192.168.0.1 activate
752 neighbor 192.168.0.2 activate
764 With this model, you don't need to have all vxlan on all nodes.
765 This model will also be needed to route traffic to an external router.
767 The symmetric model routes and bridges on both the ingress and the egress leafs.
768 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
769 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
770 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
771 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
773 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
775 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
781 net.ipv4.ip_forward=1
782 net.ipv6.conf.all.forwarding=1
783 #disable reverse path filtering
784 net.ipv4.conf.default.rp_filter=0
785 net.ipv4.conf.all.rp_filter=0
796 iface eno1 inet manual
800 iface vmbr0 inet static
802 netmask 255.255.255.0
808 iface vxlan2 inet manual
809 vxlan-local-tunnelip 192.168.0.1
811 bridge-arp-nd-suppress on
812 bridge-unicast-flood off
813 bridge-multicast-flood off
816 iface vmbr2 inet static
821 netmask 255.255.255.0
822 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
826 iface vxlan3 inet manual
827 vxlan-local-tunnelip 192.168.0.1
829 bridge-arp-nd-suppress on
830 bridge-unicast-flood off
831 bridge-multicast-flood off
834 iface vmbr3 inet static
839 netmask 255.255.255.0
840 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
843 #interconnect vxlan-vfr l3vni
845 iface vxlan4000 inet manual
846 vxlan-local-tunnelip 192.168.0.1
848 bridge-arp-nd-suppress on
849 bridge-unicast-flood off
850 bridge-multicast-flood off
854 iface vmbr4000 inet manual
855 bridge_ports vxlan4000
858 hwaddress 44:39:39:FF:40:90 #must be different on each node
869 bgp router-id 192.168.0.1
870 no bgp default ipv4-unicast
872 neighbor 192.168.0.2 remote-as 1234
873 neighbor 192.168.0.3 remote-as 1234
875 address-family l2vpn evpn
876 neighbor 192.168.0.2 activate
877 neighbor 192.168.0.3 activate
881 router bgp 1234 vrf vrf1
883 bgp router-id 192.168.0.1
885 address-family ipv4 unicast
886 redistribute connected
889 address-family l2vpn evpn
890 advertise ipv4 unicast
906 iface eno1 inet manual
910 iface vmbr0 inet static
912 netmask 255.255.255.0
918 iface vxlan2 inet manual
919 vxlan-local-tunnelip 192.168.0.2
921 bridge-arp-nd-suppress on
922 bridge-unicast-flood off
923 bridge-multicast-flood off
926 iface vmbr2 inet static
931 netmask 255.255.255.0
932 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
936 iface vxlan3 inet manual
937 vxlan-local-tunnelip 192.168.0.2
939 bridge-arp-nd-suppress on
940 bridge-unicast-flood off
941 bridge-multicast-flood off
944 iface vmbr3 inet static
949 netmask 255.255.255.0
950 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
953 #interconnect vxlan-vfr l3vni
955 iface vxlan4000 inet manual
956 vxlan-local-tunnelip 192.168.0.2
958 bridge-arp-nd-suppress on
959 bridge-unicast-flood off
960 bridge-multicast-flood off
964 iface vmbr4000 inet manual
965 bridge_ports vxlan4000
968 hwaddress 44:39:39:FF:40:91 #must be different on each node
980 bgp router-id 192.168.0.2
981 no bgp default ipv4-unicast
983 neighbor 192.168.0.1 remote-as 1234
984 neighbor 192.168.0.3 remote-as 1234
986 address-family l2vpn evpn
987 neighbor 192.168.0.1 activate
988 neighbor 192.168.0.3 activate
992 router bgp 1234 vrf vrf1
994 bgp router-id 192.168.0.2
996 address-family ipv4 unicast
997 redistribute connected
1000 address-family l2vpn evpn
1001 advertise ipv4 unicast
1017 iface eno1 inet manual
1021 iface vmbr0 inet static
1023 netmask 255.255.255.0
1029 iface vxlan2 inet manual
1030 vxlan-local-tunnelip 192.168.0.3
1032 bridge-arp-nd-suppress on
1033 bridge-unicast-flood off
1034 bridge-multicast-flood off
1037 iface vmbr2 inet static
1042 netmask 255.255.255.0
1043 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1047 iface vxlan3 inet manual
1048 vxlan-local-tunnelip 192.168.0.3
1050 bridge-arp-nd-suppress on
1051 bridge-unicast-flood off
1052 bridge-multicast-flood off
1055 iface vmbr3 inet static
1060 netmask 255.255.255.0
1061 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1064 #interconnect vxlan-vfr l3vni
1066 iface vxlan4000 inet manual
1067 vxlan-local-tunnelip 192.168.0.3
1069 bridge-arp-nd-suppress on
1070 bridge-unicast-flood off
1071 bridge-multicast-flood off
1075 iface vmbr4000 inet manual
1076 bridge_ports vxlan4000
1079 hwaddress 44:39:39:FF:40:92 #must be different on each node
1091 bgp router-id 192.168.0.3
1092 no bgp default ipv4-unicast
1094 neighbor 192.168.0.1 remote-as 1234
1095 neighbor 192.168.0.2 remote-as 1234
1097 address-family l2vpn evpn
1098 neighbor 192.168.0.1 activate
1099 neighbor 192.168.0.2 activate
1103 router bgp 1234 vrf vrf1
1105 bgp router-id 192.168.0.3
1107 address-family ipv4 unicast
1108 redistribute connected
1111 address-family l2vpn evpn
1112 advertise ipv4 unicast
1119 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1120 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1121 Routing to outside need the symmetric model.
1125 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1126 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1127 and announce this default gw to other proxmox nodes.
1138 iface eno1 inet manual
1142 iface vmbr0 inet static
1144 netmask 255.255.255.0
1152 netmask 255.255.255.0
1154 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1155 #if you have multiple external routers, you can use ecmp balancing
1156 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1159 iface vxlan2 inet manual
1160 vxlan-local-tunnelip 192.168.0.1
1162 bridge-arp-nd-suppress on
1163 bridge-unicast-flood off
1164 bridge-multicast-flood off
1167 iface vmbr2 inet static
1172 netmask 255.255.255.0
1173 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1177 iface vxlan3 inet manual
1178 vxlan-local-tunnelip 192.168.0.1
1180 bridge-arp-nd-suppress on
1181 bridge-unicast-flood off
1182 bridge-multicast-flood off
1185 iface vmbr3 inet static
1190 netmask 255.255.255.0
1191 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1194 #interconnect vxlan-vfr l3vni
1196 iface vxlan4000 inet manual
1197 vxlan-local-tunnelip 192.168.0.1
1199 bridge-arp-nd-suppress on
1200 bridge-unicast-flood off
1201 bridge-multicast-flood off
1204 iface vmbr4000 inet manual
1205 bridge_ports vxlan4000
1208 hwaddress 44:39:39:FF:40:90 #must be different on each node
1220 bgp router-id 192.168.0.1
1221 no bgp default ipv4-unicast
1223 neighbor 192.168.0.2 remote-as 1234
1224 neighbor 192.168.0.3 remote-as 1234
1226 address-family l2vpn evpn
1227 neighbor 192.168.0.2 activate
1228 neighbor 192.168.0.3 activate
1232 router bgp 1234 vrf vrf1
1234 bgp router-id 172.16.0.1
1236 address-family ipv4 unicast
1237 redistribute connected
1238 redistribute kernel !announce your default gw to all nodes
1241 address-family l2vpn evpn
1242 advertise ipv4 unicast
1258 iface eno1 inet manual
1262 iface vmbr0 inet static
1264 netmask 255.255.255.0
1270 iface vxlan2 inet manual
1271 vxlan-local-tunnelip 192.168.0.2
1273 bridge-arp-nd-suppress on
1274 bridge-unicast-flood off
1275 bridge-multicast-flood off
1278 iface vmbr2 inet static
1283 netmask 255.255.255.0
1284 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1288 iface vxlan3 inet manual
1289 vxlan-local-tunnelip 192.168.0.2
1291 bridge-arp-nd-suppress on
1292 bridge-unicast-flood off
1293 bridge-multicast-flood off
1296 iface vmbr3 inet static
1301 netmask 255.255.255.0
1302 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1305 #interconnect vxlan-vfr l3vni
1307 iface vxlan4000 inet manual
1308 vxlan-local-tunnelip 192.168.0.2
1310 bridge-arp-nd-suppress on
1311 bridge-unicast-flood off
1312 bridge-multicast-flood off
1316 iface vmbr4000 inet manual
1317 bridge_ports vxlan4000
1320 hwaddress 44:39:39:FF:40:91 #must be different on each node
1332 bgp router-id 192.168.0.2
1333 no bgp default ipv4-unicast
1335 neighbor 192.168.0.1 remote-as 1234
1336 neighbor 192.168.0.3 remote-as 1234
1338 address-family l2vpn evpn
1339 neighbor 192.168.0.1 activate
1340 neighbor 192.168.0.3 activate
1344 router bgp 1234 vrf vrf1
1346 bgp router-id 192.168.0.2
1348 address-family ipv4 unicast
1349 redistribute connected
1352 address-family l2vpn evpn
1353 advertise ipv4 unicast
1369 iface eno1 inet manual
1373 iface vmbr0 inet static
1375 netmask 255.255.255.0
1381 iface vxlan2 inet manual
1382 vxlan-local-tunnelip 192.168.0.3
1384 bridge-arp-nd-suppress on
1385 bridge-unicast-flood off
1386 bridge-multicast-flood off
1389 iface vmbr2 inet static
1394 netmask 255.255.255.0
1395 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1399 iface vxlan3 inet manual
1400 vxlan-local-tunnelip 192.168.0.3
1402 bridge-arp-nd-suppress on
1403 bridge-unicast-flood off
1404 bridge-multicast-flood off
1407 iface vmbr3 inet static
1412 netmask 255.255.255.0
1413 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1416 #interconnect vxlan-vfr l3vni
1418 iface vxlan4000 inet manual
1419 vxlan-local-tunnelip 192.168.0.3
1421 bridge-arp-nd-suppress on
1422 bridge-unicast-flood off
1423 bridge-multicast-flood off
1427 iface vmbr4000 inet manual
1428 bridge_ports vxlan4000
1431 hwaddress 44:39:39:FF:40:92 #must be different on each node
1443 bgp router-id 192.168.0.3
1444 no bgp default ipv4-unicast
1446 neighbor 192.168.0.1 remote-as 1234
1447 neighbor 192.168.0.2 remote-as 1234
1449 address-family l2vpn evpn
1450 neighbor 192.168.0.1 activate
1451 neighbor 192.168.0.2 activate
1455 router bgp 1234 vrf vrf1
1457 bgp router-id 192.168.0.3
1459 address-family ipv4 unicast
1460 redistribute connected
1463 address-family l2vpn evpn
1464 advertise ipv4 unicast
1471 multiple gateway nodes
1472 ^^^^^^^^^^^^^^^^^^^^^^
1473 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1474 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1475 and announce this default gw.
1476 The external router have ecmp routes to all proxmox nodes.(balancing).
1477 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1478 vxlan the packet to final destination.
1488 iface eno1 inet manual
1492 iface vmbr0 inet static
1494 netmask 255.255.255.0
1502 netmask 255.255.255.0
1505 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1506 #if you have multiple external routers, you can use ecmp balancing
1507 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1510 iface vxlan2 inet manual
1511 vxlan-local-tunnelip 192.168.0.1
1513 bridge-arp-nd-suppress on
1514 bridge-unicast-flood off
1515 bridge-multicast-flood off
1518 iface vmbr2 inet static
1523 netmask 255.255.255.0
1524 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1528 iface vxlan3 inet manual
1529 vxlan-local-tunnelip 192.168.0.1
1531 bridge-arp-nd-suppress on
1532 bridge-unicast-flood off
1533 bridge-multicast-flood off
1536 iface vmbr3 inet static
1541 netmask 255.255.255.0
1542 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1545 #interconnect vxlan-vfr l3vni
1547 iface vxlan4000 inet manual
1548 vxlan-local-tunnelip 192.168.0.1
1550 bridge-arp-nd-suppress on
1551 bridge-unicast-flood off
1552 bridge-multicast-flood off
1555 iface vmbr4000 inet manual
1556 bridge_ports vxlan4000
1559 hwaddress 44:39:39:FF:40:90 #must be different on each node
1571 bgp router-id 192.168.0.1
1572 no bgp default ipv4-unicast
1574 neighbor 192.168.0.2 remote-as 1234
1575 neighbor 192.168.0.3 remote-as 1234
1577 address-family l2vpn evpn
1578 neighbor 192.168.0.2 activate
1579 neighbor 192.168.0.3 activate
1583 router bgp 1234 vrf vrf1
1585 bgp router-id 172.16.0.1
1587 address-family ipv4 unicast
1588 redistribute connected
1589 redistribute kernel !announce your default gw to all nodes
1592 address-family l2vpn evpn
1593 advertise ipv4 unicast
1609 iface eno1 inet manual
1613 iface vmbr0 inet static
1615 netmask 255.255.255.0
1623 netmask 255.255.255.0
1626 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1627 #if you have multiple external routers, you can use ecmp balancing
1628 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1631 iface vxlan2 inet manual
1632 vxlan-local-tunnelip 192.168.0.2
1634 bridge-arp-nd-suppress on
1635 bridge-unicast-flood off
1636 bridge-multicast-flood off
1639 iface vmbr2 inet static
1644 netmask 255.255.255.0
1645 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1649 iface vxlan3 inet manual
1650 vxlan-local-tunnelip 192.168.0.2
1652 bridge-arp-nd-suppress on
1653 bridge-unicast-flood off
1654 bridge-multicast-flood off
1657 iface vmbr3 inet static
1662 netmask 255.255.255.0
1663 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1666 #interconnect vxlan-vfr l3vni
1668 iface vxlan4000 inet manual
1669 vxlan-local-tunnelip 192.168.0.2
1671 bridge-arp-nd-suppress on
1672 bridge-unicast-flood off
1673 bridge-multicast-flood off
1677 iface vmbr4000 inet manual
1678 bridge_ports vxlan4000
1681 hwaddress 44:39:39:FF:40:91 #must be different on each node
1693 bgp router-id 192.168.0.2
1694 no bgp default ipv4-unicast
1696 neighbor 192.168.0.1 remote-as 1234
1697 neighbor 192.168.0.3 remote-as 1234
1699 address-family l2vpn evpn
1700 neighbor 192.168.0.1 activate
1701 neighbor 192.168.0.3 activate
1705 router bgp 1234 vrf vrf1
1707 bgp router-id 172.16.0.2
1709 address-family ipv4 unicast
1710 redistribute connected
1711 redistribute kernel !announce your default gw to all nodes
1714 address-family l2vpn evpn
1715 advertise ipv4 unicast
1731 iface eno1 inet manual
1735 iface vmbr0 inet static
1737 netmask 255.255.255.0
1745 netmask 255.255.255.0
1748 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1749 #if you have multiple external routers, you can use ecmp balancing
1750 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1753 iface vxlan2 inet manual
1754 vxlan-local-tunnelip 192.168.0.3
1756 bridge-arp-nd-suppress on
1757 bridge-unicast-flood off
1758 bridge-multicast-flood off
1761 iface vmbr2 inet static
1766 netmask 255.255.255.0
1767 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1771 iface vxlan3 inet manual
1772 vxlan-local-tunnelip 192.168.0.3
1774 bridge-arp-nd-suppress on
1775 bridge-unicast-flood off
1776 bridge-multicast-flood off
1779 iface vmbr3 inet static
1784 netmask 255.255.255.0
1785 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1788 #interconnect vxlan-vfr l3vni
1790 iface vxlan4000 inet manual
1791 vxlan-local-tunnelip 192.168.0.3
1793 bridge-arp-nd-suppress on
1794 bridge-unicast-flood off
1795 bridge-multicast-flood off
1799 iface vmbr4000 inet manual
1800 bridge_ports vxlan4000
1803 hwaddress 44:39:39:FF:40:92 #must be different on each node
1815 bgp router-id 192.168.0.3
1816 no bgp default ipv4-unicast
1818 neighbor 192.168.0.1 remote-as 1234
1819 neighbor 192.168.0.2 remote-as 1234
1821 address-family l2vpn evpn
1822 neighbor 192.168.0.1 activate
1823 neighbor 192.168.0.2 activate
1827 router bgp 1234 vrf vrf1
1829 bgp router-id 172.16.0.3
1831 address-family ipv4 unicast
1832 redistribute connected
1833 redistribute kernel !announce your default gw to all nodes
1836 address-family l2vpn evpn
1837 advertise ipv4 unicast
1847 If your external router don't support ecmp to reach multiple proxmox nodes,
1848 you can setup an HA floating vip on proxmox nodes with vrrp
1850 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1851 Node1 is the primary and failover to node2 in case of failure.
1860 netmask 255.255.255.0
1863 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1866 vrrp-virtual-ip 172.16.0.10
1875 netmask 255.255.255.0
1878 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1881 vrrp-virtual-ip 172.16.0.10