4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
30 This scenario relies in head end replication, meaning that end host in case
31 of not having any entry for the destination MAC address will send out an ARP
32 to other devices / VTEPs in the VXLAN network.
33 This is done by sending the request to the VXLAN multicast group,
34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
41 iface eno1 inet manual
45 iface vmbr0 inet static
53 iface vxlan2 inet manual
55 vxlan-svcnodeip 225.20.1.1
59 iface vmbr2 inet manual
65 iface vxlan3 inet manual
67 vxlan-svcnodeip 225.20.1.1
71 iface vmbr3 inet manual
82 iface eno1 inet manual
86 iface vmbr0 inet static
94 iface vxlan2 inet manual
96 vxlan-svcnodeip 225.20.1.1
100 iface vmbr2 inet manual
107 iface vxlan3 inet manual
109 vxlan-svcnodeip 225.20.1.1
113 iface vmbr3 inet manual
124 iface eno1 inet manual
128 iface vmbr0 inet static
130 netmask 255.255.255.0
136 iface vxlan2 inet manual
138 vxlan-svcnodeip 225.20.1.1
142 iface vmbr2 inet manual
149 iface vxlan3 inet manual
151 vxlan-svcnodeip 225.20.1.1
155 iface vmbr3 inet manual
165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
166 The VXLAN is defined without a remote multicast group.
167 Instead, all the remote VTEPs are associated with the all-zero address:
168 a BUM frame will be duplicated to all these destinations.
169 The VXLAN device will still learn remote addresses automatically using source-address learning.
175 iface eno1 inet manual
179 iface vmbr0 inet static
181 netmask 255.255.255.0
188 iface vxlan2 inet manual
190 vxlan_remoteip 192.168.0.2
191 vxlan_remoteip 192.168.0.3
195 iface vmbr2 inet manual
202 iface vxlan2 inet manual
204 vxlan_remoteip 192.168.0.2
205 vxlan_remoteip 192.168.0.3
209 iface vmbr3 inet manual
220 iface eno1 inet manual
224 iface vmbr0 inet static
226 netmask 255.255.255.0
232 iface vxlan2 inet manual
234 vxlan_remoteip 192.168.0.1
235 vxlan_remoteip 192.168.0.3
240 iface vmbr2 inet manual
246 iface vxlan2 inet manual
248 vxlan_remoteip 192.168.0.1
249 vxlan_remoteip 192.168.0.3
253 iface vmbr3 inet manual
264 iface eno1 inet manual
268 iface vmbr0 inet static
270 netmask 255.255.255.0
276 iface vxlan2 inet manual
278 vxlan_remoteip 192.168.0.2
279 vxlan_remoteip 192.168.0.3
284 iface vmbr2 inet manual
290 iface vxlan2 inet manual
292 vxlan_remoteip 192.168.0.2
293 vxlan_remoteip 192.168.0.3
297 iface vmbr3 inet manual
307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
310 The control plane used here is FRR, a bgp routing software.
311 Each node in the proxmox cluster peer with each others nodes.
312 For bigger networks, or multiple proxmox clusters,
313 it's possible to use external bgp route reflector servers.
319 iface eno1 inet manual
323 iface vmbr0 inet static
325 netmask 255.255.255.0
331 iface vxlan2 inet manual
333 vxlan-local-tunnelip 192.168.0.1
335 bridge-arp-nd-suppress on
336 bridge-unicast-flood off
337 bridge-multicast-flood off
341 iface vmbr2 inet manual
348 iface vxlan3 inet manual
350 vxlan-local-tunnelip 192.168.0.1
352 bridge-arp-nd-suppress on
353 bridge-unicast-flood off
354 bridge-multicast-flood off
358 iface vmbr3 inet manual
369 no bgp default ipv4-unicast
371 neighbor 192.168.0.2 remote-as 1234
372 neighbor 192.168.0.3 remote-as 1234
374 address-family l2vpn evpn
375 neighbor 192.168.0.2 activate
376 neighbor 192.168.0.3 activate
389 iface eno1 inet manual
393 iface vmbr0 inet static
395 netmask 255.255.255.0
401 iface vxlan2 inet manual
403 vxlan-local-tunnelip 192.168.0.2
405 bridge-arp-nd-suppress on
406 bridge-unicast-flood off
407 bridge-multicast-flood off
411 iface vmbr2 inet manual
417 iface vxlan3 inet manual
419 vxlan-local-tunnelip 192.168.0.2
421 bridge-arp-nd-suppress on
422 bridge-unicast-flood off
423 bridge-multicast-flood off
427 iface vmbr3 inet manual
438 no bgp default ipv4-unicast
440 neighbor 192.168.0.1 remote-as 1234
441 neighbor 192.168.0.3 remote-as 1234
443 address-family l2vpn evpn
444 neighbor 192.168.0.1 activate
445 neighbor 192.168.0.3 activate
458 iface eno1 inet manual
462 iface vmbr0 inet static
464 netmask 255.255.255.0
470 iface vxlan2 inet manual
472 vxlan-local-tunnelip 192.168.0.3
474 bridge-arp-nd-suppress on
475 bridge-unicast-flood off
476 bridge-multicast-flood off
480 iface vmbr2 inet manual
486 iface vxlan3 inet manual
488 vxlan-local-tunnelip 192.168.0.3
490 bridge-arp-nd-suppress on
491 bridge-unicast-flood off
492 bridge-multicast-flood off
496 iface vmbr3 inet manual
508 no bgp default ipv4-unicast
510 neighbor 192.168.0.1 remote-as 1234
511 neighbor 192.168.0.2 remote-as 1234
513 address-family l2vpn evpn
514 neighbor 192.168.0.1 activate
515 neighbor 192.168.0.2 activate
523 VXLAN layer3 routing with anycast gateway
524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
526 With this need, each vmbr bridge will be the gateway for the vm.
527 Same vmbr on different node, will have same ip address and same mac address,
528 to have working vm live migration and no network disruption.
530 VXLAN layer3 routing only work with FRR and non-aware bridge.
531 (vlan aware bridge support is buggy currently).
536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
539 but only bridging on the egress.
540 This results in bi-directional VXLAN traffic traveling on different VNIs
541 in each direction (always the destination VNI) across the routed infrastructure.
543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
549 iface eno1 inet manual
553 iface vmbr0 inet static
555 netmask 255.255.255.0
561 iface vxlan2 inet manual
563 vxlan-local-tunnelip 192.168.0.1
565 bridge-arp-nd-suppress on
566 bridge-unicast-flood off
567 bridge-multicast-flood off
571 iface vmbr2 inet static
573 netmask 255.255.255.0
574 hwaddress 44:39:39:FF:40:94
583 iface vxlan3 inet manual
585 vxlan-local-tunnelip 192.168.0.1
587 bridge-arp-nd-suppress on
588 bridge-unicast-flood off
589 bridge-multicast-flood off
593 iface vmbr3 inet static
595 netmask 255.255.255.0
596 hwaddress 44:39:39:FF:40:94
610 bgp router-id 192.168.0.1
611 no bgp default ipv4-unicast
613 neighbor 192.168.0.2 remote-as 1234
614 neighbor 192.168.0.3 remote-as 1234
616 address-family l2vpn evpn
617 neighbor 192.168.0.2 activate
618 neighbor 192.168.0.3 activate
631 iface eno1 inet manual
635 iface vmbr0 inet static
637 netmask 255.255.255.0
643 iface vxlan2 inet manual
645 vxlan-local-tunnelip 192.168.0.2
647 bridge-arp-nd-suppress on
648 bridge-unicast-flood off
649 bridge-multicast-flood off
653 iface vmbr2 inet static
655 netmask 255.255.255.0
656 hwaddress 44:39:39:FF:40:94
666 iface vxlan3 inet manual
668 vxlan-local-tunnelip 192.168.0.2
670 bridge-arp-nd-suppress on
671 bridge-unicast-flood off
672 bridge-multicast-flood off
676 iface vmbr3 inet static
678 netmask 255.255.255.0
679 hwaddress 44:39:39:FF:40:94
693 bgp router-id 192.168.0.2
694 no bgp default ipv4-unicast
696 neighbor 192.168.0.1 remote-as 1234
697 neighbor 192.168.0.3 remote-as 1234
699 address-family l2vpn evpn
700 neighbor 192.168.0.1 activate
701 neighbor 192.168.0.3 activate
714 iface eno1 inet manual
718 iface vmbr0 inet static
720 netmask 255.255.255.0
726 iface vxlan2 inet manual
728 vxlan-local-tunnelip 192.168.0.3
730 bridge-arp-nd-suppress on
731 bridge-unicast-flood off
732 bridge-multicast-flood off
736 iface vmbr2 inet static
738 netmask 255.255.255.0
739 hwaddress 44:39:39:FF:40:94
748 iface vxlan3 inet manual
750 vxlan-local-tunnelip 192.168.0.3
752 bridge-arp-nd-suppress on
753 bridge-unicast-flood off
754 bridge-multicast-flood off
757 iface vmbr3 inet static
759 netmask 255.255.255.0
760 hwaddress 44:39:39:FF:40:94
774 bgp router-id 192.168.0.3
775 no bgp default ipv4-unicast
777 neighbor 192.168.0.1 remote-as 1234
778 neighbor 192.168.0.2 remote-as 1234
780 address-family l2vpn evpn
781 neighbor 192.168.0.1 activate
782 neighbor 192.168.0.2 activate
794 With this model, you don't need to have all vxlan on all nodes.
795 This model will also be needed to route traffic to an external router.
797 The symmetric model routes and bridges on both the ingress and the egress leafs.
798 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
799 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
800 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
801 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
803 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
805 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
815 iface eno1 inet manual
819 iface vmbr0 inet static
821 netmask 255.255.255.0
827 iface vxlan2 inet manual
829 vxlan-local-tunnelip 192.168.0.1
831 bridge-arp-nd-suppress on
832 bridge-unicast-flood off
833 bridge-multicast-flood off
836 iface vmbr2 inet static
841 netmask 255.255.255.0
842 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
849 iface vxlan3 inet manual
851 vxlan-local-tunnelip 192.168.0.1
853 bridge-arp-nd-suppress on
854 bridge-unicast-flood off
855 bridge-multicast-flood off
858 iface vmbr3 inet static
863 netmask 255.255.255.0
864 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
870 #interconnect vxlan-vfr l3vni
872 iface vxlan4000 inet manual
874 vxlan-local-tunnelip 192.168.0.1
876 bridge-arp-nd-suppress on
877 bridge-unicast-flood off
878 bridge-multicast-flood off
882 iface vmbr4000 inet manual
883 bridge_ports vxlan4000
897 bgp router-id 192.168.0.1
898 no bgp default ipv4-unicast
900 neighbor 192.168.0.2 remote-as 1234
901 neighbor 192.168.0.3 remote-as 1234
903 address-family l2vpn evpn
904 neighbor 192.168.0.2 activate
905 neighbor 192.168.0.3 activate
922 iface eno1 inet manual
926 iface vmbr0 inet static
928 netmask 255.255.255.0
934 iface vxlan2 inet manual
936 vxlan-local-tunnelip 192.168.0.2
938 bridge-arp-nd-suppress on
939 bridge-unicast-flood off
940 bridge-multicast-flood off
943 iface vmbr2 inet static
948 netmask 255.255.255.0
949 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
956 iface vxlan3 inet manual
958 vxlan-local-tunnelip 192.168.0.2
960 bridge-arp-nd-suppress on
961 bridge-unicast-flood off
962 bridge-multicast-flood off
965 iface vmbr3 inet static
970 netmask 255.255.255.0
971 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
977 #interconnect vxlan-vfr l3vni
979 iface vxlan4000 inet manual
981 vxlan-local-tunnelip 192.168.0.2
983 bridge-arp-nd-suppress on
984 bridge-unicast-flood off
985 bridge-multicast-flood off
989 iface vmbr4000 inet manual
990 bridge_ports vxlan4000
1005 bgp router-id 192.168.0.2
1006 no bgp default ipv4-unicast
1008 neighbor 192.168.0.1 remote-as 1234
1009 neighbor 192.168.0.3 remote-as 1234
1011 address-family l2vpn evpn
1012 neighbor 192.168.0.1 activate
1013 neighbor 192.168.0.3 activate
1030 iface eno1 inet manual
1034 iface vmbr0 inet static
1036 netmask 255.255.255.0
1042 iface vxlan2 inet manual
1044 vxlan-local-tunnelip 192.168.0.3
1046 bridge-arp-nd-suppress on
1047 bridge-unicast-flood off
1048 bridge-multicast-flood off
1051 iface vmbr2 inet static
1056 netmask 255.255.255.0
1057 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1064 iface vxlan3 inet manual
1066 vxlan-local-tunnelip 192.168.0.3
1068 bridge-arp-nd-suppress on
1069 bridge-unicast-flood off
1070 bridge-multicast-flood off
1073 iface vmbr3 inet static
1078 netmask 255.255.255.0
1079 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1085 #interconnect vxlan-vfr l3vni
1087 iface vxlan4000 inet manual
1089 vxlan-local-tunnelip 192.168.0.3
1091 bridge-arp-nd-suppress on
1092 bridge-unicast-flood off
1093 bridge-multicast-flood off
1097 iface vmbr4000 inet manual
1098 bridge_ports vxlan4000
1113 bgp router-id 192.168.0.3
1114 no bgp default ipv4-unicast
1116 neighbor 192.168.0.1 remote-as 1234
1117 neighbor 192.168.0.2 remote-as 1234
1119 address-family l2vpn evpn
1120 neighbor 192.168.0.1 activate
1121 neighbor 192.168.0.2 activate
1129 VXLAN layer3 routing with anycast gateway + routing to outside with external router with static default gw
1130 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1131 Routing to outside need the symmetric model.
1135 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1136 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1147 iface eno1 inet manual
1151 iface vmbr0 inet static
1153 netmask 255.255.255.0
1154 gateway 192.168.0.254
1162 iface vxlan2 inet manual
1164 vxlan-local-tunnelip 192.168.0.1
1166 bridge-arp-nd-suppress on
1167 bridge-unicast-flood off
1168 bridge-multicast-flood off
1171 iface vmbr2 inet static
1176 netmask 255.255.255.0
1177 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1184 iface vxlan3 inet manual
1186 vxlan-local-tunnelip 192.168.0.1
1188 bridge-arp-nd-suppress on
1189 bridge-unicast-flood off
1190 bridge-multicast-flood off
1193 iface vmbr3 inet static
1198 netmask 255.255.255.0
1199 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1205 #interconnect vxlan-vfr l3vni
1207 iface vxlan4000 inet manual
1209 vxlan-local-tunnelip 192.168.0.1
1211 bridge-arp-nd-suppress on
1212 bridge-unicast-flood off
1213 bridge-multicast-flood off
1216 iface vmbr4000 inet manual
1217 bridge_ports vxlan4000
1232 bgp router-id 192.168.0.1
1233 no bgp default ipv4-unicast
1235 neighbor 192.168.0.2 remote-as 1234
1236 neighbor 192.168.0.3 remote-as 1234
1238 address-family ipv4 unicast
1242 address-family ipv6 unicast
1246 address-family l2vpn evpn
1247 neighbor 192.168.0.2 activate
1248 neighbor 192.168.0.3 activate
1252 router bgp 1234 vrf vrf1
1254 address-family ipv4 unicast
1255 redistribute connected
1258 address-family ipv6 unicast
1259 redistribute connected
1262 address-family l2vpn evpn
1263 default-originate ipv4
1264 default-originate ipv6
1280 iface eno1 inet manual
1284 iface vmbr0 inet static
1286 netmask 255.255.255.0
1292 iface vxlan2 inet manual
1294 vxlan-local-tunnelip 192.168.0.2
1296 bridge-arp-nd-suppress on
1297 bridge-unicast-flood off
1298 bridge-multicast-flood off
1301 iface vmbr2 inet static
1306 netmask 255.255.255.0
1307 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1314 iface vxlan3 inet manual
1316 vxlan-local-tunnelip 192.168.0.2
1318 bridge-arp-nd-suppress on
1319 bridge-unicast-flood off
1320 bridge-multicast-flood off
1323 iface vmbr3 inet static
1328 netmask 255.255.255.0
1329 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1335 #interconnect vxlan-vfr l3vni
1337 iface vxlan4000 inet manual
1339 vxlan-local-tunnelip 192.168.0.2
1341 bridge-arp-nd-suppress on
1342 bridge-unicast-flood off
1343 bridge-multicast-flood off
1347 iface vmbr4000 inet manual
1348 bridge_ports vxlan4000
1363 bgp router-id 192.168.0.2
1364 no bgp default ipv4-unicast
1366 neighbor 192.168.0.1 remote-as 1234
1367 neighbor 192.168.0.3 remote-as 1234
1369 address-family l2vpn evpn
1370 neighbor 192.168.0.1 activate
1371 neighbor 192.168.0.3 activate
1388 iface eno1 inet manual
1392 iface vmbr0 inet static
1394 netmask 255.255.255.0
1400 iface vxlan2 inet manual
1402 vxlan-local-tunnelip 192.168.0.3
1404 bridge-arp-nd-suppress on
1405 bridge-unicast-flood off
1406 bridge-multicast-flood off
1409 iface vmbr2 inet static
1414 netmask 255.255.255.0
1415 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1422 iface vxlan3 inet manual
1424 vxlan-local-tunnelip 192.168.0.3
1426 bridge-arp-nd-suppress on
1427 bridge-unicast-flood off
1428 bridge-multicast-flood off
1431 iface vmbr3 inet static
1436 netmask 255.255.255.0
1437 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1443 #interconnect vxlan-vfr l3vni
1445 iface vxlan4000 inet manual
1447 vxlan-local-tunnelip 192.168.0.3
1449 bridge-arp-nd-suppress on
1450 bridge-unicast-flood off
1451 bridge-multicast-flood off
1455 iface vmbr4000 inet manual
1456 bridge_ports vxlan4000
1471 bgp router-id 192.168.0.3
1472 no bgp default ipv4-unicast
1474 neighbor 192.168.0.1 remote-as 1234
1475 neighbor 192.168.0.2 remote-as 1234
1477 address-family l2vpn evpn
1478 neighbor 192.168.0.1 activate
1479 neighbor 192.168.0.2 activate
1487 multiple gateway nodes
1488 ^^^^^^^^^^^^^^^^^^^^^^
1489 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1490 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1491 and announce this default gw in the vrf (default originate)
1492 The external router have ecmp routes to all proxmox nodes.(balancing).
1493 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1494 vxlan the packet to final destination.
1496 If you have multiple gateway nodes, disable rp_filter as packet could incoming in a 1 node, and outgoing
1501 net.ipv4.conf.default.rp_filter=0
1502 net.ipv4.conf.all.rp_filter=0
1514 iface eno1 inet manual
1518 iface vmbr0 inet static
1520 netmask 255.255.255.0
1521 gateway 192.168.0.254
1529 iface vxlan2 inet manual
1531 vxlan-local-tunnelip 192.168.0.1
1533 bridge-arp-nd-suppress on
1534 bridge-unicast-flood off
1535 bridge-multicast-flood off
1538 iface vmbr2 inet static
1543 netmask 255.255.255.0
1544 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1551 iface vxlan3 inet manual
1553 vxlan-local-tunnelip 192.168.0.1
1555 bridge-arp-nd-suppress on
1556 bridge-unicast-flood off
1557 bridge-multicast-flood off
1560 iface vmbr3 inet static
1565 netmask 255.255.255.0
1566 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1572 #interconnect vxlan-vfr l3vni
1574 iface vxlan4000 inet manual
1576 vxlan-local-tunnelip 192.168.0.1
1578 bridge-arp-nd-suppress on
1579 bridge-unicast-flood off
1580 bridge-multicast-flood off
1583 iface vmbr4000 inet manual
1584 bridge_ports vxlan4000
1599 bgp router-id 192.168.0.1
1600 no bgp default ipv4-unicast
1602 neighbor 192.168.0.2 remote-as 1234
1603 neighbor 192.168.0.3 remote-as 1234
1605 address-family ipv4 unicast
1609 address-family ipv6 unicast
1613 address-family l2vpn evpn
1614 neighbor 192.168.0.2 activate
1615 neighbor 192.168.0.3 activate
1619 router bgp 1234 vrf vrf1
1621 address-family ipv4 unicast
1622 redistribute connected
1625 address-family ipv6 unicast
1626 redistribute connected
1629 address-family l2vpn evpn
1630 default-originate ipv4
1631 default-originate ipv6
1647 iface eno1 inet manual
1651 iface vmbr0 inet static
1653 netmask 255.255.255.0
1654 gateway 192.168.0.254
1662 iface vxlan2 inet manual
1664 vxlan-local-tunnelip 192.168.0.2
1666 bridge-arp-nd-suppress on
1667 bridge-unicast-flood off
1668 bridge-multicast-flood off
1671 iface vmbr2 inet static
1676 netmask 255.255.255.0
1677 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1684 iface vxlan3 inet manual
1686 vxlan-local-tunnelip 192.168.0.2
1688 bridge-arp-nd-suppress on
1689 bridge-unicast-flood off
1690 bridge-multicast-flood off
1693 iface vmbr3 inet static
1698 netmask 255.255.255.0
1699 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1705 #interconnect vxlan-vfr l3vni
1707 iface vxlan4000 inet manual
1709 vxlan-local-tunnelip 192.168.0.2
1711 bridge-arp-nd-suppress on
1712 bridge-unicast-flood off
1713 bridge-multicast-flood off
1717 iface vmbr4000 inet manual
1718 bridge_ports vxlan4000
1733 bgp router-id 192.168.0.2
1734 no bgp default ipv4-unicast
1736 neighbor 192.168.0.1 remote-as 1234
1737 neighbor 192.168.0.3 remote-as 1234
1739 address-family ipv4 unicast
1743 address-family ipv6 unicast
1747 address-family l2vpn evpn
1748 neighbor 192.168.0.1 activate
1749 neighbor 192.168.0.3 activate
1753 address-family ipv4 unicast
1754 redistribute connected
1757 address-family ipv6 unicast
1758 redistribute connected
1761 address-family l2vpn evpn
1762 default-originate ipv4
1763 default-originate ipv6
1779 iface eno1 inet manual
1783 iface vmbr0 inet static
1785 netmask 255.255.255.0
1786 gateway 192.168.0.254
1794 iface vxlan2 inet manual
1796 vxlan-local-tunnelip 192.168.0.3
1798 bridge-arp-nd-suppress on
1799 bridge-unicast-flood off
1800 bridge-multicast-flood off
1803 iface vmbr2 inet static
1808 netmask 255.255.255.0
1809 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1816 iface vxlan3 inet manual
1818 vxlan-local-tunnelip 192.168.0.3
1820 bridge-arp-nd-suppress on
1821 bridge-unicast-flood off
1822 bridge-multicast-flood off
1825 iface vmbr3 inet static
1830 netmask 255.255.255.0
1831 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1837 #interconnect vxlan-vfr l3vni
1839 iface vxlan4000 inet manual
1841 vxlan-local-tunnelip 192.168.0.3
1843 bridge-arp-nd-suppress on
1844 bridge-unicast-flood off
1845 bridge-multicast-flood off
1849 iface vmbr4000 inet manual
1850 bridge_ports vxlan4000
1865 bgp router-id 192.168.0.3
1866 no bgp default ipv4-unicast
1868 neighbor 192.168.0.1 remote-as 1234
1869 neighbor 192.168.0.2 remote-as 1234
1871 address-family ipv4 unicast
1875 address-family ipv6 unicast
1879 address-family l2vpn evpn
1880 neighbor 192.168.0.1 activate
1881 neighbor 192.168.0.2 activate
1885 router bgp 1234 vrf vrf1
1887 address-family ipv4 unicast
1888 redistribute connected
1891 address-family ipv6 unicast
1892 redistribute connected
1895 address-family l2vpn evpn
1896 default-originate ipv4
1897 default-originate ipv6
1907 If your external router doesn't support 'ECMP static routes' to reach multiple
1908 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1909 Virtual Router Redundancy Protocol (VRRP).
1911 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1912 Node1 is the primary with failover to node2 in case of outage.
1914 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1915 #TODO : It should be possible to do it with frr directly with last version.
1921 iface vmbr0 inet static
1923 netmask 255.255.255.0
1924 gateway 192.168.0.254
1930 vrrp-virtual-ip 192.168.0.10
1937 iface vmbr0 inet static
1939 netmask 255.255.255.0
1940 gateway 192.168.0.254
1946 vrrp-virtual-ip 192.168.0.10
1951 gateway node(s) with a upstream bgp router
1952 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1953 Setup is almost the same than with a static gateway, but we'll connect to an upstream bgp router.
1955 example with node1 as gateway (192.168.0.1) for evpn-bgp, and an upstream bgp router (running frr too) 192.168.0.254.
1966 bgp router-id 192.168.0.1
1967 no bgp default ipv4-unicast
1969 neighbor 192.168.0.2 remote-as 1234
1970 neighbor 192.168.0.3 remote-as 1234
1971 neighbor 192.168.0.254 remote-as external
1973 address-family ipv4 unicast
1975 neighbor 192.168.0.254 activate
1978 address-family ipv6 unicast
1980 neighbor 192.168.0.254 activate
1983 address-family l2vpn evpn
1984 neighbor 192.168.0.1 activate
1985 neighbor 192.168.0.2 activate
1986 neighbor 192.168.0.254 activate
1990 router bgp 1234 vrf vrf1
1992 address-family ipv4 unicast
1993 redistribute connected
1996 address-family ipv6 unicast
1997 redistribute connected
2000 address-family l2vpn evpn
2001 default-originate ipv4
2002 default-originate ipv6
2013 ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24
2014 ip prefix-list NO32 seq 20 deny any
2017 bgp router-id 192.168.0.254
2018 bgp bestpath as-path multipath-relax
2019 neighbor 192.168.0.1 remote-as external
2020 neighbor 192.168.0.1 capability extended-nexthop
2022 address-family ipv4 unicast
2023 neighbor 192.168.0.1 default-originate
2024 neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
2027 address-family ipv6 unicast
2028 neighbor 192.168.0.1 default-originate
2029 neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
2037 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
2038 to avoid that all node peers with each others nodes.
2039 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
2040 single point of failure, a minimum of two servers acting as an RR is highly
2041 recommended for redundancy.
2043 Below is an example of configuration with 'frr', with `rrserver1
2044 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
2049 bgp router-id 192.168.0.200
2050 bgp cluster-id 1.1.1.1 #cluster-id must be the same on each route reflector
2051 bgp log-neighbor-changes
2052 no bgp default ipv4-unicast
2053 neighbor fabric peer-group
2054 neighbor fabric remote-as 1234
2055 neighbor fabric capability extended-nexthop
2056 neighbor fabric update-source 192.168.0.200
2057 bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
2059 address-family l2vpn evpn
2060 neighbor fabric activate
2061 neighbor fabric route-reflector-client
2062 neighbor fabric allowas-in
2072 bgp router-id 192.168.0.201
2073 bgp cluster-id 1.1.1.1
2074 bgp log-neighbor-changes
2075 no bgp default ipv4-unicast
2076 neighbor fabric peer-group
2077 neighbor fabric remote-as 1234
2078 neighbor fabric capability extended-nexthop
2079 neighbor fabric update-source 192.168.0.201
2080 bgp listen range 192.168.0.0/24 peer-group fabric
2082 address-family l2vpn evpn
2083 neighbor fabric activate
2084 neighbor fabric route-reflector-client
2085 neighbor fabric allowas-in
2095 bgp router-id 192.168.0.x
2096 no bgp default ipv4-unicast
2098 neighbor 192.168.0.200 remote-as 1234
2099 neighbor 192.168.0.201 remote-as 1234
2101 address-family l2vpn evpn
2102 neighbor 192.168.0.200 activate
2103 neighbor 192.168.0.201 activate