4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
30 This scenario relies in head end replication, meaning that end host in case
31 of not having any entry for the destination MAC address will send out an ARP
32 to other devices / VTEPs in the VXLAN network.
33 This is done by sending the request to the VXLAN multicast group,
34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
41 iface eno1 inet manual
45 iface vmbr0 inet static
53 iface vxlan2 inet manual
55 vxlan-svcnodeip 225.20.1.1
59 iface vmbr2 inet manual
65 iface vxlan3 inet manual
67 vxlan-svcnodeip 225.20.1.1
71 iface vmbr3 inet manual
82 iface eno1 inet manual
86 iface vmbr0 inet static
94 iface vxlan2 inet manual
96 vxlan-svcnodeip 225.20.1.1
100 iface vmbr2 inet manual
107 iface vxlan3 inet manual
109 vxlan-svcnodeip 225.20.1.1
113 iface vmbr3 inet manual
124 iface eno1 inet manual
128 iface vmbr0 inet static
130 netmask 255.255.255.0
136 iface vxlan2 inet manual
138 vxlan-svcnodeip 225.20.1.1
142 iface vmbr2 inet manual
149 iface vxlan3 inet manual
151 vxlan-svcnodeip 225.20.1.1
155 iface vmbr3 inet manual
165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
166 The VXLAN is defined without a remote multicast group.
167 Instead, all the remote VTEPs are associated with the all-zero address:
168 a BUM frame will be duplicated to all these destinations.
169 The VXLAN device will still learn remote addresses automatically using source-address learning.
175 iface eno1 inet manual
179 iface vmbr0 inet static
181 netmask 255.255.255.0
188 iface vxlan2 inet manual
190 vxlan_remoteip 192.168.0.2
191 vxlan_remoteip 192.168.0.3
195 iface vmbr2 inet manual
202 iface vxlan2 inet manual
204 vxlan_remoteip 192.168.0.2
205 vxlan_remoteip 192.168.0.3
209 iface vmbr3 inet manual
220 iface eno1 inet manual
224 iface vmbr0 inet static
226 netmask 255.255.255.0
232 iface vxlan2 inet manual
234 vxlan_remoteip 192.168.0.1
235 vxlan_remoteip 192.168.0.3
240 iface vmbr2 inet manual
246 iface vxlan2 inet manual
248 vxlan_remoteip 192.168.0.1
249 vxlan_remoteip 192.168.0.3
253 iface vmbr3 inet manual
264 iface eno1 inet manual
268 iface vmbr0 inet static
270 netmask 255.255.255.0
276 iface vxlan2 inet manual
278 vxlan_remoteip 192.168.0.2
279 vxlan_remoteip 192.168.0.3
284 iface vmbr2 inet manual
290 iface vxlan2 inet manual
292 vxlan_remoteip 192.168.0.2
293 vxlan_remoteip 192.168.0.3
297 iface vmbr3 inet manual
307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
310 The control plane used here is FRR, a bgp routing software.
311 Each node in the proxmox cluster peer with each others nodes.
312 For bigger networks, or multiple proxmox clusters,
313 it's possible to use external bgp route reflector servers.
319 iface eno1 inet manual
323 iface vmbr0 inet static
325 netmask 255.255.255.0
331 iface vxlan2 inet manual
333 vxlan-local-tunnelip 192.168.0.1
335 bridge-arp-nd-suppress on
336 bridge-unicast-flood off
337 bridge-multicast-flood off
341 iface vmbr2 inet manual
348 iface vxlan3 inet manual
350 vxlan-local-tunnelip 192.168.0.1
352 bridge-arp-nd-suppress on
353 bridge-unicast-flood off
354 bridge-multicast-flood off
358 iface vmbr3 inet manual
369 no bgp default ipv4-unicast
371 neighbor 192.168.0.2 remote-as 1234
372 neighbor 192.168.0.3 remote-as 1234
374 address-family l2vpn evpn
375 neighbor 192.168.0.2 activate
376 neighbor 192.168.0.3 activate
389 iface eno1 inet manual
393 iface vmbr0 inet static
395 netmask 255.255.255.0
401 iface vxlan2 inet manual
403 vxlan-local-tunnelip 192.168.0.2
405 bridge-arp-nd-suppress on
406 bridge-unicast-flood off
407 bridge-multicast-flood off
411 iface vmbr2 inet manual
417 iface vxlan3 inet manual
419 vxlan-local-tunnelip 192.168.0.2
421 bridge-arp-nd-suppress on
422 bridge-unicast-flood off
423 bridge-multicast-flood off
427 iface vmbr3 inet manual
438 no bgp default ipv4-unicast
440 neighbor 192.168.0.1 remote-as 1234
441 neighbor 192.168.0.3 remote-as 1234
443 address-family l2vpn evpn
444 neighbor 192.168.0.1 activate
445 neighbor 192.168.0.3 activate
458 iface eno1 inet manual
462 iface vmbr0 inet static
464 netmask 255.255.255.0
470 iface vxlan2 inet manual
472 vxlan-local-tunnelip 192.168.0.3
474 bridge-arp-nd-suppress on
475 bridge-unicast-flood off
476 bridge-multicast-flood off
480 iface vmbr2 inet manual
486 iface vxlan3 inet manual
488 vxlan-local-tunnelip 192.168.0.3
490 bridge-arp-nd-suppress on
491 bridge-unicast-flood off
492 bridge-multicast-flood off
496 iface vmbr3 inet manual
508 no bgp default ipv4-unicast
510 neighbor 192.168.0.1 remote-as 1234
511 neighbor 192.168.0.2 remote-as 1234
513 address-family l2vpn evpn
514 neighbor 192.168.0.1 activate
515 neighbor 192.168.0.2 activate
523 VXLAN layer3 routing with anycast gateway
524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
526 With this need, each vmbr bridge will be the gateway for the vm.
527 Same vmbr on different node, will have same ip address and same mac address,
528 to have working vm live migration and no network disruption.
530 VXLAN layer3 routing only work with FRR and non-aware bridge.
531 (vlan aware bridge support is buggy currently).
536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
539 but only bridging on the egress.
540 This results in bi-directional VXLAN traffic traveling on different VNIs
541 in each direction (always the destination VNI) across the routed infrastructure.
543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
550 net.ipv4.ip_forward=1
551 net.ipv6.conf.all.forwarding=1
558 iface eno1 inet manual
562 iface vmbr0 inet static
564 netmask 255.255.255.0
570 iface vxlan2 inet manual
572 vxlan-local-tunnelip 192.168.0.1
574 bridge-arp-nd-suppress on
575 bridge-unicast-flood off
576 bridge-multicast-flood off
580 iface vmbr2 inet static
582 netmask 255.255.255.0
583 hwaddress 44:39:39:FF:40:94
590 iface vxlan3 inet manual
592 vxlan-local-tunnelip 192.168.0.1
594 bridge-arp-nd-suppress on
595 bridge-unicast-flood off
596 bridge-multicast-flood off
600 iface vmbr3 inet static
602 netmask 255.255.255.0
603 hwaddress 44:39:39:FF:40:94
614 bgp router-id 192.168.0.1
615 no bgp default ipv4-unicast
617 neighbor 192.168.0.2 remote-as 1234
618 neighbor 192.168.0.3 remote-as 1234
620 address-family l2vpn evpn
621 neighbor 192.168.0.2 activate
622 neighbor 192.168.0.3 activate
635 iface eno1 inet manual
639 iface vmbr0 inet static
641 netmask 255.255.255.0
647 iface vxlan2 inet manual
649 vxlan-local-tunnelip 192.168.0.2
651 bridge-arp-nd-suppress on
652 bridge-unicast-flood off
653 bridge-multicast-flood off
657 iface vmbr2 inet static
659 netmask 255.255.255.0
660 hwaddress 44:39:39:FF:40:94
667 iface vxlan3 inet manual
669 vxlan-local-tunnelip 192.168.0.2
671 bridge-arp-nd-suppress on
672 bridge-unicast-flood off
673 bridge-multicast-flood off
677 iface vmbr3 inet static
679 netmask 255.255.255.0
680 hwaddress 44:39:39:FF:40:94
691 bgp router-id 192.168.0.2
692 no bgp default ipv4-unicast
694 neighbor 192.168.0.1 remote-as 1234
695 neighbor 192.168.0.3 remote-as 1234
697 address-family l2vpn evpn
698 neighbor 192.168.0.1 activate
699 neighbor 192.168.0.3 activate
712 iface eno1 inet manual
716 iface vmbr0 inet static
718 netmask 255.255.255.0
724 iface vxlan2 inet manual
726 vxlan-local-tunnelip 192.168.0.3
728 bridge-arp-nd-suppress on
729 bridge-unicast-flood off
730 bridge-multicast-flood off
734 iface vmbr2 inet static
736 netmask 255.255.255.0
737 hwaddress 44:39:39:FF:40:94
744 iface vxlan3 inet manual
746 vxlan-local-tunnelip 192.168.0.3
748 bridge-arp-nd-suppress on
749 bridge-unicast-flood off
750 bridge-multicast-flood off
754 iface vmbr3 inet static
756 netmask 255.255.255.0
757 hwaddress 44:39:39:FF:40:94
768 bgp router-id 192.168.0.3
769 no bgp default ipv4-unicast
771 neighbor 192.168.0.1 remote-as 1234
772 neighbor 192.168.0.2 remote-as 1234
774 address-family l2vpn evpn
775 neighbor 192.168.0.1 activate
776 neighbor 192.168.0.2 activate
788 With this model, you don't need to have all vxlan on all nodes.
789 This model will also be needed to route traffic to an external router.
791 The symmetric model routes and bridges on both the ingress and the egress leafs.
792 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
793 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
794 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
795 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
797 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
799 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
805 net.ipv4.ip_forward=1
806 net.ipv6.conf.all.forwarding=1
817 iface eno1 inet manual
821 iface vmbr0 inet static
823 netmask 255.255.255.0
829 iface vxlan2 inet manual
831 vxlan-local-tunnelip 192.168.0.1
833 bridge-arp-nd-suppress on
834 bridge-unicast-flood off
835 bridge-multicast-flood off
838 iface vmbr2 inet static
843 netmask 255.255.255.0
844 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
848 iface vxlan3 inet manual
850 vxlan-local-tunnelip 192.168.0.1
852 bridge-arp-nd-suppress on
853 bridge-unicast-flood off
854 bridge-multicast-flood off
857 iface vmbr3 inet static
862 netmask 255.255.255.0
863 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
866 #interconnect vxlan-vfr l3vni
868 iface vxlan4000 inet manual
870 vxlan-local-tunnelip 192.168.0.1
872 bridge-arp-nd-suppress on
873 bridge-unicast-flood off
874 bridge-multicast-flood off
878 iface vmbr4000 inet manual
879 bridge_ports vxlan4000
893 bgp router-id 192.168.0.1
894 no bgp default ipv4-unicast
896 neighbor 192.168.0.2 remote-as 1234
897 neighbor 192.168.0.3 remote-as 1234
899 address-family l2vpn evpn
900 neighbor 192.168.0.2 activate
901 neighbor 192.168.0.3 activate
918 iface eno1 inet manual
922 iface vmbr0 inet static
924 netmask 255.255.255.0
930 iface vxlan2 inet manual
932 vxlan-local-tunnelip 192.168.0.2
934 bridge-arp-nd-suppress on
935 bridge-unicast-flood off
936 bridge-multicast-flood off
939 iface vmbr2 inet static
944 netmask 255.255.255.0
945 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
949 iface vxlan3 inet manual
951 vxlan-local-tunnelip 192.168.0.2
953 bridge-arp-nd-suppress on
954 bridge-unicast-flood off
955 bridge-multicast-flood off
958 iface vmbr3 inet static
963 netmask 255.255.255.0
964 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
967 #interconnect vxlan-vfr l3vni
969 iface vxlan4000 inet manual
971 vxlan-local-tunnelip 192.168.0.2
973 bridge-arp-nd-suppress on
974 bridge-unicast-flood off
975 bridge-multicast-flood off
979 iface vmbr4000 inet manual
980 bridge_ports vxlan4000
995 bgp router-id 192.168.0.2
996 no bgp default ipv4-unicast
998 neighbor 192.168.0.1 remote-as 1234
999 neighbor 192.168.0.3 remote-as 1234
1001 address-family l2vpn evpn
1002 neighbor 192.168.0.1 activate
1003 neighbor 192.168.0.3 activate
1020 iface eno1 inet manual
1024 iface vmbr0 inet static
1026 netmask 255.255.255.0
1032 iface vxlan2 inet manual
1034 vxlan-local-tunnelip 192.168.0.3
1036 bridge-arp-nd-suppress on
1037 bridge-unicast-flood off
1038 bridge-multicast-flood off
1041 iface vmbr2 inet static
1046 netmask 255.255.255.0
1047 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1051 iface vxlan3 inet manual
1053 vxlan-local-tunnelip 192.168.0.3
1055 bridge-arp-nd-suppress on
1056 bridge-unicast-flood off
1057 bridge-multicast-flood off
1060 iface vmbr3 inet static
1065 netmask 255.255.255.0
1066 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1069 #interconnect vxlan-vfr l3vni
1071 iface vxlan4000 inet manual
1073 vxlan-local-tunnelip 192.168.0.3
1075 bridge-arp-nd-suppress on
1076 bridge-unicast-flood off
1077 bridge-multicast-flood off
1081 iface vmbr4000 inet manual
1082 bridge_ports vxlan4000
1097 bgp router-id 192.168.0.3
1098 no bgp default ipv4-unicast
1100 neighbor 192.168.0.1 remote-as 1234
1101 neighbor 192.168.0.2 remote-as 1234
1103 address-family l2vpn evpn
1104 neighbor 192.168.0.1 activate
1105 neighbor 192.168.0.2 activate
1113 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1114 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1115 Routing to outside need the symmetric model.
1119 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1120 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1131 iface eno1 inet manual
1135 iface vmbr0 inet static
1137 netmask 255.255.255.0
1138 gateway 192.168.0.254
1144 iface vxlan2 inet manual
1146 vxlan-local-tunnelip 192.168.0.1
1148 bridge-arp-nd-suppress on
1149 bridge-unicast-flood off
1150 bridge-multicast-flood off
1153 iface vmbr2 inet static
1158 netmask 255.255.255.0
1159 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1163 iface vxlan3 inet manual
1165 vxlan-local-tunnelip 192.168.0.1
1167 bridge-arp-nd-suppress on
1168 bridge-unicast-flood off
1169 bridge-multicast-flood off
1172 iface vmbr3 inet static
1177 netmask 255.255.255.0
1178 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1181 #interconnect vxlan-vfr l3vni
1183 iface vxlan4000 inet manual
1185 vxlan-local-tunnelip 192.168.0.1
1187 bridge-arp-nd-suppress on
1188 bridge-unicast-flood off
1189 bridge-multicast-flood off
1192 iface vmbr4000 inet manual
1193 bridge_ports vxlan4000
1208 bgp router-id 192.168.0.1
1209 no bgp default ipv4-unicast
1211 neighbor 192.168.0.2 remote-as 1234
1212 neighbor 192.168.0.3 remote-as 1234
1214 address-family ipv4 unicast
1218 address-family l2vpn evpn
1219 neighbor 192.168.0.2 activate
1220 neighbor 192.168.0.3 activate
1224 router bgp 1234 vrf vrf1
1226 address-family l2vpn evpn
1227 default-originate ipv4
1243 iface eno1 inet manual
1247 iface vmbr0 inet static
1249 netmask 255.255.255.0
1255 iface vxlan2 inet manual
1257 vxlan-local-tunnelip 192.168.0.2
1259 bridge-arp-nd-suppress on
1260 bridge-unicast-flood off
1261 bridge-multicast-flood off
1264 iface vmbr2 inet static
1269 netmask 255.255.255.0
1270 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1274 iface vxlan3 inet manual
1276 vxlan-local-tunnelip 192.168.0.2
1278 bridge-arp-nd-suppress on
1279 bridge-unicast-flood off
1280 bridge-multicast-flood off
1283 iface vmbr3 inet static
1288 netmask 255.255.255.0
1289 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1292 #interconnect vxlan-vfr l3vni
1294 iface vxlan4000 inet manual
1296 vxlan-local-tunnelip 192.168.0.2
1298 bridge-arp-nd-suppress on
1299 bridge-unicast-flood off
1300 bridge-multicast-flood off
1304 iface vmbr4000 inet manual
1305 bridge_ports vxlan4000
1320 bgp router-id 192.168.0.2
1321 no bgp default ipv4-unicast
1323 neighbor 192.168.0.1 remote-as 1234
1324 neighbor 192.168.0.3 remote-as 1234
1326 address-family l2vpn evpn
1327 neighbor 192.168.0.1 activate
1328 neighbor 192.168.0.3 activate
1345 iface eno1 inet manual
1349 iface vmbr0 inet static
1351 netmask 255.255.255.0
1357 iface vxlan2 inet manual
1359 vxlan-local-tunnelip 192.168.0.3
1361 bridge-arp-nd-suppress on
1362 bridge-unicast-flood off
1363 bridge-multicast-flood off
1366 iface vmbr2 inet static
1371 netmask 255.255.255.0
1372 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1376 iface vxlan3 inet manual
1378 vxlan-local-tunnelip 192.168.0.3
1380 bridge-arp-nd-suppress on
1381 bridge-unicast-flood off
1382 bridge-multicast-flood off
1385 iface vmbr3 inet static
1390 netmask 255.255.255.0
1391 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1394 #interconnect vxlan-vfr l3vni
1396 iface vxlan4000 inet manual
1398 vxlan-local-tunnelip 192.168.0.3
1400 bridge-arp-nd-suppress on
1401 bridge-unicast-flood off
1402 bridge-multicast-flood off
1406 iface vmbr4000 inet manual
1407 bridge_ports vxlan4000
1422 bgp router-id 192.168.0.3
1423 no bgp default ipv4-unicast
1425 neighbor 192.168.0.1 remote-as 1234
1426 neighbor 192.168.0.2 remote-as 1234
1428 address-family l2vpn evpn
1429 neighbor 192.168.0.1 activate
1430 neighbor 192.168.0.2 activate
1438 multiple gateway nodes
1439 ^^^^^^^^^^^^^^^^^^^^^^
1440 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1441 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1442 and announce this default gw in the vrf (default originate)
1443 The external router have ecmp routes to all proxmox nodes.(balancing).
1444 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1445 vxlan the packet to final destination.
1455 iface eno1 inet manual
1459 iface vmbr0 inet static
1461 netmask 255.255.255.0
1462 gateway 192.168.0.254
1468 iface vxlan2 inet manual
1470 vxlan-local-tunnelip 192.168.0.1
1472 bridge-arp-nd-suppress on
1473 bridge-unicast-flood off
1474 bridge-multicast-flood off
1477 iface vmbr2 inet static
1482 netmask 255.255.255.0
1483 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1487 iface vxlan3 inet manual
1489 vxlan-local-tunnelip 192.168.0.1
1491 bridge-arp-nd-suppress on
1492 bridge-unicast-flood off
1493 bridge-multicast-flood off
1496 iface vmbr3 inet static
1501 netmask 255.255.255.0
1502 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1505 #interconnect vxlan-vfr l3vni
1507 iface vxlan4000 inet manual
1509 vxlan-local-tunnelip 192.168.0.1
1511 bridge-arp-nd-suppress on
1512 bridge-unicast-flood off
1513 bridge-multicast-flood off
1516 iface vmbr4000 inet manual
1517 bridge_ports vxlan4000
1532 bgp router-id 192.168.0.1
1533 no bgp default ipv4-unicast
1535 neighbor 192.168.0.2 remote-as 1234
1536 neighbor 192.168.0.3 remote-as 1234
1538 address-family ipv4 unicast
1542 address-family l2vpn evpn
1543 neighbor 192.168.0.2 activate
1544 neighbor 192.168.0.3 activate
1548 router bgp 1234 vrf vrf1
1550 address-family l2vpn evpn
1551 default-originate ipv4
1567 iface eno1 inet manual
1571 iface vmbr0 inet static
1573 netmask 255.255.255.0
1574 gateway 192.168.0.254
1580 iface vxlan2 inet manual
1582 vxlan-local-tunnelip 192.168.0.2
1584 bridge-arp-nd-suppress on
1585 bridge-unicast-flood off
1586 bridge-multicast-flood off
1589 iface vmbr2 inet static
1594 netmask 255.255.255.0
1595 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1599 iface vxlan3 inet manual
1601 vxlan-local-tunnelip 192.168.0.2
1603 bridge-arp-nd-suppress on
1604 bridge-unicast-flood off
1605 bridge-multicast-flood off
1608 iface vmbr3 inet static
1613 netmask 255.255.255.0
1614 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1617 #interconnect vxlan-vfr l3vni
1619 iface vxlan4000 inet manual
1621 vxlan-local-tunnelip 192.168.0.2
1623 bridge-arp-nd-suppress on
1624 bridge-unicast-flood off
1625 bridge-multicast-flood off
1629 iface vmbr4000 inet manual
1630 bridge_ports vxlan4000
1645 bgp router-id 192.168.0.2
1646 no bgp default ipv4-unicast
1648 neighbor 192.168.0.1 remote-as 1234
1649 neighbor 192.168.0.3 remote-as 1234
1651 address-family ipv4 unicast
1655 address-family l2vpn evpn
1656 neighbor 192.168.0.1 activate
1657 neighbor 192.168.0.3 activate
1661 address-family l2vpn evpn
1662 default-originate ipv4
1678 iface eno1 inet manual
1682 iface vmbr0 inet static
1684 netmask 255.255.255.0
1685 gateway 192.168.0.254
1691 iface vxlan2 inet manual
1693 vxlan-local-tunnelip 192.168.0.3
1695 bridge-arp-nd-suppress on
1696 bridge-unicast-flood off
1697 bridge-multicast-flood off
1700 iface vmbr2 inet static
1705 netmask 255.255.255.0
1706 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1710 iface vxlan3 inet manual
1712 vxlan-local-tunnelip 192.168.0.3
1714 bridge-arp-nd-suppress on
1715 bridge-unicast-flood off
1716 bridge-multicast-flood off
1719 iface vmbr3 inet static
1724 netmask 255.255.255.0
1725 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1728 #interconnect vxlan-vfr l3vni
1730 iface vxlan4000 inet manual
1732 vxlan-local-tunnelip 192.168.0.3
1734 bridge-arp-nd-suppress on
1735 bridge-unicast-flood off
1736 bridge-multicast-flood off
1740 iface vmbr4000 inet manual
1741 bridge_ports vxlan4000
1756 bgp router-id 192.168.0.3
1757 no bgp default ipv4-unicast
1759 neighbor 192.168.0.1 remote-as 1234
1760 neighbor 192.168.0.2 remote-as 1234
1762 address-family ipv4 unicast
1766 address-family l2vpn evpn
1767 neighbor 192.168.0.1 activate
1768 neighbor 192.168.0.2 activate
1772 router bgp 1234 vrf vrf1
1774 address-family l2vpn evpn
1775 default-originate ipv4
1785 If your external router don't support ecmp static route to reach multiple proxmox nodes,
1786 you can setup an HA floating vip on proxmox nodes with vrrp
1788 In this example, we will setup an floating 192.168.0.10 ip on node1 and node2.
1789 Node1 is the primary and failover to node2 in case of failure.
1791 This setup need vrrpd package (apt install vrrpd).
1792 #TODO : It should be possible to do it with frr directly with last version.
1798 iface vmbr0 inet static
1800 netmask 255.255.255.0
1801 gateway 192.168.0.254
1807 vrrp-virtual-ip 192.168.0.10
1814 iface vmbr0 inet static
1816 netmask 255.255.255.0
1817 gateway 192.168.0.254
1823 vrrp-virtual-ip 192.168.0.10
1829 If you have a lot of proxmox nodes, or multiple proxmox clusters,
1830 maybe do you want to avoid that each node peer with each others nodes.
1831 For this, you can create dedicated route reflectors servers. (Minimum 2 servers for redundancy).
1832 Here an example of configuration with frr, with rrserver1 (192.168.0.200) and rrserver2 (192.168.0.201).
1838 bgp router-id 192.168.0.200
1839 bgp cluster-id 1.1.1.1 #cluster-id must be the same on each route reflector
1840 bgp log-neighbor-changes
1841 no bgp default ipv4-unicast
1842 neighbor fabric peer-group
1843 neighbor fabric remote-as 1234
1844 neighbor fabric capability extended-nexthop
1845 neighbor fabric update-source 192.168.0.200
1846 bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
1848 address-family l2vpn evpn
1849 neighbor fabric activate
1850 neighbor fabric route-reflector-client
1851 neighbor fabric allowas-in
1861 bgp router-id 192.168.0.201
1862 bgp cluster-id 1.1.1.1
1863 bgp log-neighbor-changes
1864 no bgp default ipv4-unicast
1865 neighbor fabric peer-group
1866 neighbor fabric remote-as 1234
1867 neighbor fabric capability extended-nexthop
1868 neighbor fabric update-source 192.168.0.201
1869 bgp listen range 192.168.0.0/24 peer-group fabric
1871 address-family l2vpn evpn
1872 neighbor fabric activate
1873 neighbor fabric route-reflector-client
1874 neighbor fabric allowas-in
1884 bgp router-id 192.168.0.x
1885 no bgp default ipv4-unicast
1887 neighbor 192.168.0.200 remote-as 1234
1888 neighbor 192.168.0.201 remote-as 1234
1890 address-family ipv4 unicast
1894 address-family l2vpn evpn
1895 neighbor 192.168.0.200 activate
1896 neighbor 192.168.0.201 activate
1902 #TODO : Documentation with bgp upstream router.