vxlan-evpn : add documentation to external routing
[pve-docs.git] / vxlan-and-evpn.adoc
1
2 ////
3
4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
7
8 ////
9
10
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
13
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
18
19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
21
22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
23
24 multicast mode
25 ^^^^^^^^^^^^^^
26
27 This scenario relies in head end replication, meaning that end host in case
28 of not having any entry for the destination MAC address will send out an ARP
29 to other devices / VTEPs in the VXLAN network.
30 This is done by sending the request to the VXLAN multicast group,
31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
32
33
34 * node1
35
36 ----
37 auto eno1
38 iface eno1 inet manual
39
40 auto vmbr0
41 iface vmbr0 inet static
42 address 192.168.0.1
43 netmask 255.255.255.0
44 bridge_ports eno1
45 bridge_stp off
46 bridge_fd 0
47
48 auto vxlan2
49 iface vxlan2 inet manual
50 vxlan-svcnodeip 225.20.1.1
51 vxlan-physdev eno1
52
53 auto vmbr2
54 iface vmbr2 inet manual
55 bridge_ports vxlan2
56 bridge_stp off
57 bridge_fd 0
58
59 auto vxlan3
60 iface vxlan3 inet manual
61 vxlan-svcnodeip 225.20.1.1
62 vxlan-physdev eno1
63
64 auto vmbr3
65 iface vmbr3 inet manual
66 bridge_ports vxlan3
67 bridge_stp off
68 bridge_fd 0
69 ----
70
71
72 * node2
73
74 ----
75 auto eno1
76 iface eno1 inet manual
77
78 auto vmbr0
79 iface vmbr0 inet static
80 address 192.168.0.2
81 netmask 255.255.255.0
82 bridge_ports eno1
83 bridge_stp off
84 bridge_fd 0
85
86 auto vxlan2
87 iface vxlan2 inet manual
88 vxlan-svcnodeip 225.20.1.1
89 vxlan-physdev eno1
90
91 auto vmbr2
92 iface vmbr2 inet manual
93 bridge_ports vxlan2
94 bridge_stp off
95 bridge_fd 0
96
97
98 auto vxlan3
99 iface vxlan3 inet manual
100 vxlan-svcnodeip 225.20.1.1
101 vxlan-physdev eno1
102
103 auto vmbr3
104 iface vmbr3 inet manual
105 bridge_ports vxlan3
106 bridge_stp off
107 bridge_fd 0
108 ----
109
110
111 * node3
112
113 ----
114 auto eno1
115 iface eno1 inet manual
116
117 auto vmbr0
118 iface vmbr0 inet static
119 address 192.168.0.3
120 netmask 255.255.255.0
121 bridge_ports eno1
122 bridge_stp off
123 bridge_fd 0
124
125 auto vxlan2
126 iface vxlan2 inet manual
127 vxlan-svcnodeip 225.20.1.1
128 vxlan-physdev eno1
129
130 auto vmbr2
131 iface vmbr2 inet manual
132 bridge_ports vxlan2
133 bridge_stp off
134 bridge_fd 0
135
136
137 auto vxlan3
138 iface vxlan3 inet manual
139 vxlan-svcnodeip 225.20.1.1
140 vxlan-physdev eno1
141
142 auto vmbr3
143 iface vmbr3 inet manual
144 bridge_ports vxlan3
145 bridge_stp off
146 bridge_fd 0
147 ----
148
149
150 unicast mode
151 ^^^^^^^^^^^^
152
153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
154 The VXLAN is defined without a remote multicast group.
155 Instead, all the remote VTEPs are associated with the all-zero address:
156 a BUM frame will be duplicated to all these destinations.
157 The VXLAN device will still learn remote addresses automatically using source-address learning.
158
159 * node1
160
161 ----
162 auto eno1
163 iface eno1 inet manual
164
165 auto vmbr0
166 iface vmbr0 inet static
167 address 192.168.0.1
168 netmask 255.255.255.0
169 bridge_ports eno1
170 bridge_stp off
171 bridge_fd 0
172
173
174 auto vxlan2
175 iface vxlan2 inet manual
176 vxlan_remoteip 192.168.0.2
177 vxlan_remoteip 192.168.0.3
178
179
180 auto vmbr2
181 iface vmbr2 inet manual
182 bridge_ports vxlan2
183 bridge_stp off
184 bridge_fd 0
185
186
187 auto vxlan3
188 iface vxlan2 inet manual
189 vxlan_remoteip 192.168.0.2
190 vxlan_remoteip 192.168.0.3
191
192
193 auto vmbr3
194 iface vmbr3 inet manual
195 bridge_ports vxlan3
196 bridge_stp off
197 bridge_fd 0
198 ----
199
200
201 * node2
202
203 ----
204 auto eno1
205 iface eno1 inet manual
206
207 auto vmbr0
208 iface vmbr0 inet static
209 address 192.168.0.2
210 netmask 255.255.255.0
211 bridge_ports eno1
212 bridge_stp off
213 bridge_fd 0
214
215 auto vxlan2
216 iface vxlan2 inet manual
217 vxlan_remoteip 192.168.0.1
218 vxlan_remoteip 192.168.0.3
219
220
221
222 auto vmbr2
223 iface vmbr2 inet manual
224 bridge_ports vxlan2
225 bridge_stp off
226 bridge_fd 0
227
228 auto vxlan3
229 iface vxlan2 inet manual
230 vxlan_remoteip 192.168.0.1
231 vxlan_remoteip 192.168.0.3
232
233
234 auto vmbr3
235 iface vmbr3 inet manual
236 bridge_ports vxlan3
237 bridge_stp off
238 bridge_fd 0
239 ----
240
241
242 * node3
243
244 ----
245 auto eno1
246 iface eno1 inet manual
247
248 auto vmbr0
249 iface vmbr0 inet static
250 address 192.168.0.3
251 netmask 255.255.255.0
252 bridge_ports eno1
253 bridge_stp off
254 bridge_fd 0
255
256 auto vxlan2
257 iface vxlan2 inet manual
258 vxlan_remoteip 192.168.0.2
259 vxlan_remoteip 192.168.0.3
260
261
262
263 auto vmbr2
264 iface vmbr2 inet manual
265 bridge_ports vxlan2
266 bridge_stp off
267 bridge_fd 0
268
269 auto vxlan3
270 iface vxlan2 inet manual
271 vxlan_remoteip 192.168.0.2
272 vxlan_remoteip 192.168.0.3
273
274
275 auto vmbr3
276 iface vmbr3 inet manual
277 bridge_ports vxlan3
278 bridge_stp off
279 bridge_fd 0
280 ----
281
282
283 bgp-evpn
284 ^^^^^^^^
285
286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
288
289 The control plane used here is FRR, a bgp routing software.
290 Each node in the proxmox cluster peer with each others nodes.
291 For bigger networks, or multiple proxmox clusters,
292 it's possible to use external bgp route reflector servers.
293
294 * node1
295
296 ----
297 auto eno1
298 iface eno1 inet manual
299
300 auto vmbr0
301 iface vmbr0 inet static
302 address 192.168.0.1
303 netmask 255.255.255.0
304 bridge_ports eno1
305 bridge_stp off
306 bridge_fd 0
307
308 auto vxlan2
309 iface vxlan2 inet manual
310 vxlan-local-tunnelip 192.168.0.1
311 bridge-learning off
312 bridge-arp-nd-suppress on
313 bridge-unicast-flood off
314 bridge-multicast-flood off
315
316
317 auto vmbr2
318 iface vmbr2 inet manual
319 bridge_ports vxlan2
320 bridge_stp off
321 bridge_fd 0
322
323
324 auto vxlan3
325 iface vxlan3 inet manual
326 vxlan-local-tunnelip 192.168.0.1
327 bridge-learning off
328 bridge-arp-nd-suppress on
329 bridge-unicast-flood off
330 bridge-multicast-flood off
331
332
333 auto vmbr3
334 iface vmbr3 inet manual
335 bridge_ports vxlan3
336 bridge_stp off
337 bridge_fd 0
338 ----
339
340
341 /etc/frr/frr.conf
342
343 ----
344 router bgp 1234
345 no bgp default ipv4-unicast
346 coalesce-time 1000
347 neighbor 192.168.0.2 remote-as 1234
348 neighbor 192.168.0.3 remote-as 1234
349 !
350 address-family l2vpn evpn
351 neighbor 192.168.0.2 activate
352 neighbor 192.168.0.3 activate
353 advertise-all-vni
354 exit-address-family
355 !
356 line vty
357 !
358 ----
359
360
361 * node2
362
363 ----
364 auto eno1
365 iface eno1 inet manual
366
367 auto vmbr0
368 iface vmbr0 inet static
369 address 192.168.0.2
370 netmask 255.255.255.0
371 bridge_ports eno1
372 bridge_stp off
373 bridge_fd 0
374
375 auto vxlan2
376 iface vxlan2 inet manual
377 vxlan-local-tunnelip 192.168.0.2
378 bridge-learning off
379 bridge-arp-nd-suppress on
380 bridge-unicast-flood off
381 bridge-multicast-flood off
382
383
384 auto vmbr2
385 iface vmbr2 inet manual
386 bridge_ports vxlan2
387 bridge_stp off
388 bridge_fd 0
389
390 auto vxlan3
391 iface vxlan3 inet manual
392 vxlan-local-tunnelip 192.168.0.2
393 bridge-learning off
394 bridge-arp-nd-suppress on
395 bridge-unicast-flood off
396 bridge-multicast-flood off
397
398
399 auto vmbr3
400 iface vmbr3 inet manual
401 bridge_ports vxlan3
402 bridge_stp off
403 bridge_fd 0
404 ----
405
406
407 /etc/frr/frr.conf
408
409 ----
410 router bgp 1234
411 no bgp default ipv4-unicast
412 coalesce-time 1000
413 neighbor 192.168.0.1 remote-as 1234
414 neighbor 192.168.0.3 remote-as 1234
415 !
416 address-family l2vpn evpn
417 neighbor 192.168.0.1 activate
418 neighbor 192.168.0.3 activate
419 advertise-all-vni
420 exit-address-family
421 !
422 line vty
423 !
424 ----
425
426
427 * node3
428
429 ----
430 auto eno1
431 iface eno1 inet manual
432
433 auto vmbr0
434 iface vmbr0 inet static
435 address 192.168.0.2
436 netmask 255.255.255.0
437 bridge_ports eno1
438 bridge_stp off
439 bridge_fd 0
440
441 auto vxlan2
442 iface vxlan2 inet manual
443 vxlan-local-tunnelip 192.168.0.3
444 bridge-learning off
445 bridge-arp-nd-suppress on
446 bridge-unicast-flood off
447 bridge-multicast-flood off
448
449
450 auto vmbr2
451 iface vmbr2 inet manual
452 bridge_ports vxlan2
453 bridge_stp off
454 bridge_fd 0
455
456 auto vxlan3
457 iface vxlan3 inet manual
458 vxlan-local-tunnelip 192.168.0.3
459 bridge-learning off
460 bridge-arp-nd-suppress on
461 bridge-unicast-flood off
462 bridge-multicast-flood off
463
464
465 auto vmbr3
466 iface vmbr3 inet manual
467 bridge_ports vxlan3
468 bridge_stp off
469 bridge_fd 0
470 ----
471
472
473 /etc/frr/frr.conf
474
475
476 ----
477 router bgp 1234
478 no bgp default ipv4-unicast
479 coalesce-time 1000
480 neighbor 192.168.0.1 remote-as 1234
481 neighbor 192.168.0.2 remote-as 1234
482 !
483 address-family l2vpn evpn
484 neighbor 192.168.0.1 activate
485 neighbor 192.168.0.2 activate
486 advertise-all-vni
487 exit-address-family
488 !
489 line vty
490 !
491 ----
492
493 VXLAN layer3 routing with anycast gateway
494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
495
496 With this need, each vmbr bridge will be the gateway for the vm.
497 Same vmbr on different node, will have same ip address and same mac address,
498 to have working vm live migration and no network disruption.
499
500 VXLAN layer3 routing only work with FRR and non-aware bridge.
501 (vlan aware bridge support is buggy currently).
502
503 asymmetric model
504 ^^^^^^^^^^^^^^^^
505
506 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
507
508 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
509 but only bridging on the egress.
510 This results in bi-directional VXLAN traffic traveling on different VNIs
511 in each direction (always the destination VNI) across the routed infrastructure.
512
513 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
514
515
516 sysctl.conf tuning
517
518 ----
519 #enable routing
520 net.ipv4.ip_forward=1
521 net.ipv6.conf.all.forwarding=1
522 ----
523
524 * node1
525
526 ----
527 auto eno1
528 iface eno1 inet manual
529
530 auto vmbr0
531 iface vmbr0 inet static
532 address 192.168.0.1
533 netmask 255.255.255.0
534 bridge_ports eno1
535 bridge_stp off
536 bridge_fd 0
537
538 auto vxlan2
539 iface vxlan2 inet manual
540 vxlan-local-tunnelip 192.168.0.1
541 bridge-learning off
542 bridge-arp-nd-suppress on
543 bridge-unicast-flood off
544 bridge-multicast-flood off
545
546
547 auto vmbr2
548 iface vmbr2 inet static
549 address 10.0.2.254
550 netmask 255.255.255.0
551 hwaddress 44:39:39:FF:40:94
552 bridge_ports vxlan2
553 bridge_stp off
554 bridge_fd 0
555
556
557 auto vxlan3
558 iface vxlan3 inet manual
559 vxlan-local-tunnelip 192.168.0.1
560 bridge-learning off
561 bridge-arp-nd-suppress on
562 bridge-unicast-flood off
563 bridge-multicast-flood off
564
565
566 auto vmbr3
567 iface vmbr3 inet static
568 address 10.0.3.254
569 netmask 255.255.255.0
570 hwaddress 44:39:39:FF:40:94
571 bridge_ports vxlan3
572 bridge_stp off
573 bridge_fd 0
574 ----
575
576
577 frr.conf
578
579 ----
580 router bgp 1234
581 bgp router-id 192.168.0.1
582 no bgp default ipv4-unicast
583 coalesce-time 1000
584 neighbor 192.168.0.2 remote-as 1234
585 neighbor 192.168.0.3 remote-as 1234
586 !
587 address-family l2vpn evpn
588 neighbor 192.168.0.2 activate
589 neighbor 192.168.0.3 activate
590 advertise-all-vni
591 exit-address-family
592 !
593 line vty
594 !
595 ----
596
597
598 * node2
599
600 ----
601 auto eno1
602 iface eno1 inet manual
603
604 auto vmbr0
605 iface vmbr0 inet static
606 address 192.168.0.2
607 netmask 255.255.255.0
608 bridge_ports eno1
609 bridge_stp off
610 bridge_fd 0
611
612 auto vxlan2
613 iface vxlan2 inet manual
614 vxlan-local-tunnelip 192.168.0.2
615 bridge-learning off
616 bridge-arp-nd-suppress on
617 bridge-unicast-flood off
618 bridge-multicast-flood off
619
620
621 auto vmbr2
622 iface vmbr2 inet static
623 address 10.0.2.254
624 netmask 255.255.255.0
625 hwaddress 44:39:39:FF:40:94
626 bridge_ports vxlan2
627 bridge_stp off
628 bridge_fd 0
629
630
631 auto vxlan3
632 iface vxlan3 inet manual
633 vxlan-local-tunnelip 192.168.0.2
634 bridge-learning off
635 bridge-arp-nd-suppress on
636 bridge-unicast-flood off
637 bridge-multicast-flood off
638
639
640 auto vmbr3
641 iface vmbr3 inet static
642 address 10.0.3.254
643 netmask 255.255.255.0
644 hwaddress 44:39:39:FF:40:94
645 bridge_ports vxlan3
646 bridge_stp off
647 bridge_fd 0
648 ----
649
650
651 frr.conf
652
653 ----
654 router bgp 1234
655 bgp router-id 192.168.0.2
656 no bgp default ipv4-unicast
657 coalesce-time 1000
658 neighbor 192.168.0.1 remote-as 1234
659 neighbor 192.168.0.3 remote-as 1234
660 !
661 address-family l2vpn evpn
662 neighbor 192.168.0.1 activate
663 neighbor 192.168.0.3 activate
664 advertise-all-vni
665 exit-address-family
666 !
667 line vty
668 !
669 ----
670
671
672 * node3
673
674 ----
675 auto eno1
676 iface eno1 inet manual
677
678 auto vmbr0
679 iface vmbr0 inet static
680 address 192.168.0.3
681 netmask 255.255.255.0
682 bridge_ports eno1
683 bridge_stp off
684 bridge_fd 0
685
686 auto vxlan2
687 iface vxlan2 inet manual
688 vxlan-local-tunnelip 192.168.0.3
689 bridge-learning off
690 bridge-arp-nd-suppress on
691 bridge-unicast-flood off
692 bridge-multicast-flood off
693
694
695 auto vmbr2
696 iface vmbr2 inet static
697 address 10.0.2.254
698 netmask 255.255.255.0
699 hwaddress 44:39:39:FF:40:94
700 bridge_ports vxlan2
701 bridge_stp off
702 bridge_fd 0
703
704
705 auto vxlan3
706 iface vxlan3 inet manual
707 vxlan-local-tunnelip 192.168.0.3
708 bridge-learning off
709 bridge-arp-nd-suppress on
710 bridge-unicast-flood off
711 bridge-multicast-flood off
712
713
714 auto vmbr3
715 iface vmbr3 inet static
716 address 10.0.3.254
717 netmask 255.255.255.0
718 hwaddress 44:39:39:FF:40:94
719 bridge_ports vxlan3
720 bridge_stp off
721 bridge_fd 0
722 ----
723
724
725 frr.conf
726
727 ----
728 router bgp 1234
729 bgp router-id 192.168.0.3
730 no bgp default ipv4-unicast
731 coalesce-time 1000
732 neighbor 192.168.0.1 remote-as 1234
733 neighbor 192.168.0.2 remote-as 1234
734 !
735 address-family l2vpn evpn
736 neighbor 192.168.0.1 activate
737 neighbor 192.168.0.2 activate
738 advertise-all-vni
739 exit-address-family
740 !
741 line vty
742 !
743 ----
744
745
746 symmetric model
747 ^^^^^^^^^^^^^^^
748
749 With this model, you don't need to have all vxlan on all nodes.
750 This model will also be needed to route traffic to an external router.
751
752 The symmetric model routes and bridges on both the ingress and the egress leafs.
753 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
754 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
755 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
756 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
757
758 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
759
760 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
761
762 sysctl.conf tuning
763
764 ----
765 #enable routing
766 net.ipv4.ip_forward=1
767 net.ipv6.conf.all.forwarding=1
768 #disable reverse path filtering
769 net.ipv4.conf.default.rp_filter=0
770 net.ipv4.conf.all.rp_filter=0
771 #allow frr to work with vrf (kernel >4.14 bug)
772 net.ipv4.tcp_l3mdev_accept=1
773 ----
774
775 * node1
776
777 ----
778 auto vrf1
779 iface vrf1
780 vrf-table auto
781
782 auto eno1
783 iface eno1 inet manual
784
785 auto vmbr0
786 iface vmbr0 inet static
787 address 192.168.0.1
788 netmask 255.255.255.0
789 bridge_ports eno1
790 bridge_stp off
791 bridge_fd 0
792
793 auto vxlan2
794 iface vxlan2 inet manual
795 vxlan-local-tunnelip 192.168.0.1
796 bridge-learning off
797 bridge-arp-nd-suppress on
798 bridge-unicast-flood off
799 bridge-multicast-flood off
800
801 auto vmbr2
802 iface vmbr2 inet static
803 bridge_ports vxlan2
804 bridge_stp off
805 bridge_fd 0
806 address 10.0.2.254
807 netmask 255.255.255.0
808 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
809 vrf vrf1
810
811 auto vxlan3
812 iface vxlan3 inet manual
813 vxlan-local-tunnelip 192.168.0.1
814 bridge-learning off
815 bridge-arp-nd-suppress on
816 bridge-unicast-flood off
817 bridge-multicast-flood off
818
819 auto vmbr3
820 iface vmbr3 inet static
821 bridge_ports vxlan3
822 bridge_stp off
823 bridge_fd 0
824 address 10.0.3.254
825 netmask 255.255.255.0
826 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
827 vrf vrf1
828
829 #interconnect vxlan-vfr l3vni
830 auto vxlan4000
831 iface vxlan4000 inet manual
832 vxlan-local-tunnelip 192.168.0.1
833 bridge-learning off
834 bridge-arp-nd-suppress on
835 bridge-unicast-flood off
836 bridge-multicast-flood off
837
838
839 auto vmbr4000
840 iface vmbr4000 inet manual
841 bridge_ports vxlan4000
842 bridge_stp off
843 bridge_fd 0
844 hwaddress 44:39:39:FF:40:90 #must be different on each node
845 vrf vrf1
846 ----
847
848 frr.conf
849
850 ----
851 vrf vrf1
852 vni 4000
853 !
854 router bgp 1234
855 bgp router-id 192.168.0.1
856 no bgp default ipv4-unicast
857 coalesce-time 1000
858 neighbor 192.168.0.2 remote-as 1234
859 neighbor 192.168.0.3 remote-as 1234
860 !
861 address-family l2vpn evpn
862 neighbor 192.168.0.2 activate
863 neighbor 192.168.0.3 activate
864 advertise-all-vni
865 exit-address-family
866 !
867 router bgp 1234 vrf vrf1
868 !
869 bgp router-id 192.168.0.1
870 !
871 address-family ipv4 unicast
872 redistribute connected
873 exit-address-family
874 !
875 address-family l2vpn evpn
876 advertise ipv4 unicast
877 exit-address-family
878 !
879 line vty
880 !
881 ----
882
883
884 * node2
885
886 ----
887 auto vrf1
888 iface vrf1
889 vrf-table auto
890
891 auto eno1
892 iface eno1 inet manual
893
894 auto vmbr0
895 iface vmbr0 inet static
896 address 192.168.0.2
897 netmask 255.255.255.0
898 bridge_ports eno1
899 bridge_stp off
900 bridge_fd 0
901
902 auto vxlan2
903 iface vxlan2 inet manual
904 vxlan-local-tunnelip 192.168.0.2
905 bridge-learning off
906 bridge-arp-nd-suppress on
907 bridge-unicast-flood off
908 bridge-multicast-flood off
909
910 auto vmbr2
911 iface vmbr2 inet static
912 bridge_ports vxlan2
913 bridge_stp off
914 bridge_fd 0
915 address 10.0.2.254
916 netmask 255.255.255.0
917 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
918 vrf vrf1
919
920 auto vxlan3
921 iface vxlan3 inet manual
922 vxlan-local-tunnelip 192.168.0.2
923 bridge-learning off
924 bridge-arp-nd-suppress on
925 bridge-unicast-flood off
926 bridge-multicast-flood off
927
928 auto vmbr3
929 iface vmbr3 inet static
930 bridge_ports vxlan3
931 bridge_stp off
932 bridge_fd 0
933 address 10.0.3.254
934 netmask 255.255.255.0
935 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
936 vrf vrf1
937
938 #interconnect vxlan-vfr l3vni
939 auto vxlan4000
940 iface vxlan4000 inet manual
941 vxlan-local-tunnelip 192.168.0.2
942 bridge-learning off
943 bridge-arp-nd-suppress on
944 bridge-unicast-flood off
945 bridge-multicast-flood off
946
947
948 auto vmbr4000
949 iface vmbr4000 inet manual
950 bridge_ports vxlan4000
951 bridge_stp off
952 bridge_fd 0
953 hwaddress 44:39:39:FF:40:91 #must be different on each node
954 vrf vrf1
955 ----
956
957
958 frr.conf
959
960 ----
961 vrf vrf1
962 vni 4000
963 !
964 router bgp 1234
965 bgp router-id 192.168.0.2
966 no bgp default ipv4-unicast
967 coalesce-time 1000
968 neighbor 192.168.0.1 remote-as 1234
969 neighbor 192.168.0.3 remote-as 1234
970 !
971 address-family l2vpn evpn
972 neighbor 192.168.0.1 activate
973 neighbor 192.168.0.3 activate
974 advertise-all-vni
975 exit-address-family
976 !
977 router bgp 1234 vrf vrf1
978 !
979 bgp router-id 192.168.0.2
980 !
981 address-family ipv4 unicast
982 redistribute connected
983 exit-address-family
984 !
985 address-family l2vpn evpn
986 advertise ipv4 unicast
987 exit-address-family
988 !
989 line vty
990 !
991 ----
992
993
994 * node3
995
996 ----
997 auto vrf1
998 iface vrf1
999 vrf-table auto
1000
1001 auto eno1
1002 iface eno1 inet manual
1003
1004 auto vmbr0
1005 iface vmbr0 inet static
1006 address 192.168.0.3
1007 netmask 255.255.255.0
1008 bridge_ports eno1
1009 bridge_stp off
1010 bridge_fd 0
1011
1012 auto vxlan2
1013 iface vxlan2 inet manual
1014 vxlan-local-tunnelip 192.168.0.3
1015 bridge-learning off
1016 bridge-arp-nd-suppress on
1017 bridge-unicast-flood off
1018 bridge-multicast-flood off
1019
1020 auto vmbr2
1021 iface vmbr2 inet static
1022 bridge_ports vxlan2
1023 bridge_stp off
1024 bridge_fd 0
1025 address 10.0.2.254
1026 netmask 255.255.255.0
1027 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1028 vrf vrf1
1029
1030 auto vxlan3
1031 iface vxlan3 inet manual
1032 vxlan-local-tunnelip 192.168.0.3
1033 bridge-learning off
1034 bridge-arp-nd-suppress on
1035 bridge-unicast-flood off
1036 bridge-multicast-flood off
1037
1038 auto vmbr3
1039 iface vmbr3 inet static
1040 bridge_ports vxlan3
1041 bridge_stp off
1042 bridge_fd 0
1043 address 10.0.3.254
1044 netmask 255.255.255.0
1045 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1046 vrf vrf1
1047
1048 #interconnect vxlan-vfr l3vni
1049 auto vxlan4000
1050 iface vxlan4000 inet manual
1051 vxlan-local-tunnelip 192.168.0.3
1052 bridge-learning off
1053 bridge-arp-nd-suppress on
1054 bridge-unicast-flood off
1055 bridge-multicast-flood off
1056
1057
1058 auto vmbr4000
1059 iface vmbr4000 inet manual
1060 bridge_ports vxlan4000
1061 bridge_stp off
1062 bridge_fd 0
1063 hwaddress 44:39:39:FF:40:92 #must be different on each node
1064 vrf vrf1
1065 ----
1066
1067
1068 frr.conf
1069
1070 ----
1071 vrf vrf1
1072 vni 4000
1073 !
1074 router bgp 1234
1075 bgp router-id 192.168.0.3
1076 no bgp default ipv4-unicast
1077 coalesce-time 1000
1078 neighbor 192.168.0.1 remote-as 1234
1079 neighbor 192.168.0.2 remote-as 1234
1080 !
1081 address-family l2vpn evpn
1082 neighbor 192.168.0.1 activate
1083 neighbor 192.168.0.2 activate
1084 advertise-all-vni
1085 exit-address-family
1086 !
1087 router bgp 1234 vrf vrf1
1088 !
1089 bgp router-id 192.168.0.3
1090 !
1091 address-family ipv4 unicast
1092 redistribute connected
1093 exit-address-family
1094 !
1095 address-family l2vpn evpn
1096 advertise ipv4 unicast
1097 exit-address-family
1098 !
1099 line vty
1100 !
1101 ----
1102
1103 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1104 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1105 Routing to outside need the symmetric model.
1106
1107 1 gateway node
1108 ^^^^^^^^^^^^^^
1109 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1110 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1111 and announce this default gw to other proxmox nodes.
1112
1113
1114 *node1
1115
1116 ----
1117 auto vrf1
1118 iface vrf1
1119 vrf-table auto
1120
1121 auto eno1
1122 iface eno1 inet manual
1123
1124 auto vmbr0
1125 iface vmbr0 inet static
1126 address 192.168.0.1
1127 netmask 255.255.255.0
1128 bridge_ports eno1
1129 bridge_stp off
1130 bridge_fd 0
1131
1132 auto eno2
1133 iface eno2
1134 address 172.16.0.1
1135 netmask 255.255.255.0
1136 vrf vrf1
1137 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1138 #if you have multiple external routers, you can use ecmp balancing
1139 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1140
1141 auto vxlan2
1142 iface vxlan2 inet manual
1143 vxlan-local-tunnelip 192.168.0.1
1144 bridge-learning off
1145 bridge-arp-nd-suppress on
1146 bridge-unicast-flood off
1147 bridge-multicast-flood off
1148
1149 auto vmbr2
1150 iface vmbr2 inet static
1151 bridge_ports vxlan2
1152 bridge_stp off
1153 bridge_fd 0
1154 address 10.0.2.254
1155 netmask 255.255.255.0
1156 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1157 vrf vrf1
1158
1159 auto vxlan3
1160 iface vxlan3 inet manual
1161 vxlan-local-tunnelip 192.168.0.1
1162 bridge-learning off
1163 bridge-arp-nd-suppress on
1164 bridge-unicast-flood off
1165 bridge-multicast-flood off
1166
1167 auto vmbr3
1168 iface vmbr3 inet static
1169 bridge_ports vxlan3
1170 bridge_stp off
1171 bridge_fd 0
1172 address 10.0.3.254
1173 netmask 255.255.255.0
1174 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1175 vrf vrf1
1176
1177 #interconnect vxlan-vfr l3vni
1178 auto vxlan4000
1179 iface vxlan4000 inet manual
1180 vxlan-local-tunnelip 192.168.0.1
1181 bridge-learning off
1182 bridge-arp-nd-suppress on
1183 bridge-unicast-flood off
1184 bridge-multicast-flood off
1185
1186 auto vmbr4000
1187 iface vmbr4000 inet manual
1188 bridge_ports vxlan4000
1189 bridge_stp off
1190 bridge_fd 0
1191 hwaddress 44:39:39:FF:40:90 #must be different on each node
1192 vrf vrf1
1193 ----
1194
1195
1196 frr.conf
1197
1198 ----
1199 vrf vrf1
1200 vni 4000
1201 !
1202 router bgp 1234
1203 bgp router-id 192.168.0.1
1204 no bgp default ipv4-unicast
1205 coalesce-time 1000
1206 neighbor 192.168.0.2 remote-as 1234
1207 neighbor 192.168.0.3 remote-as 1234
1208 !
1209 address-family l2vpn evpn
1210 neighbor 192.168.0.2 activate
1211 neighbor 192.168.0.3 activate
1212 advertise-all-vni
1213 exit-address-family
1214 !
1215 router bgp 1234 vrf vrf1
1216 !
1217 bgp router-id 172.16.0.1
1218 !
1219 address-family ipv4 unicast
1220 redistribute connected
1221 redistribute kernel !announce your default gw to all nodes
1222 exit-address-family
1223 !
1224 address-family l2vpn evpn
1225 advertise ipv4 unicast
1226 exit-address-family
1227 !
1228 line vty
1229 !
1230 ----
1231
1232
1233 * node2
1234
1235 ----
1236 auto vrf1
1237 iface vrf1
1238 vrf-table auto
1239
1240 auto eno1
1241 iface eno1 inet manual
1242
1243 auto vmbr0
1244 iface vmbr0 inet static
1245 address 192.168.0.2
1246 netmask 255.255.255.0
1247 bridge_ports eno1
1248 bridge_stp off
1249 bridge_fd 0
1250
1251 auto vxlan2
1252 iface vxlan2 inet manual
1253 vxlan-local-tunnelip 192.168.0.2
1254 bridge-learning off
1255 bridge-arp-nd-suppress on
1256 bridge-unicast-flood off
1257 bridge-multicast-flood off
1258
1259 auto vmbr2
1260 iface vmbr2 inet static
1261 bridge_ports vxlan2
1262 bridge_stp off
1263 bridge_fd 0
1264 address 10.0.2.254
1265 netmask 255.255.255.0
1266 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1267 vrf vrf1
1268
1269 auto vxlan3
1270 iface vxlan3 inet manual
1271 vxlan-local-tunnelip 192.168.0.2
1272 bridge-learning off
1273 bridge-arp-nd-suppress on
1274 bridge-unicast-flood off
1275 bridge-multicast-flood off
1276
1277 auto vmbr3
1278 iface vmbr3 inet static
1279 bridge_ports vxlan3
1280 bridge_stp off
1281 bridge_fd 0
1282 address 10.0.3.254
1283 netmask 255.255.255.0
1284 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1285 vrf vrf1
1286
1287 #interconnect vxlan-vfr l3vni
1288 auto vxlan4000
1289 iface vxlan4000 inet manual
1290 vxlan-local-tunnelip 192.168.0.2
1291 bridge-learning off
1292 bridge-arp-nd-suppress on
1293 bridge-unicast-flood off
1294 bridge-multicast-flood off
1295
1296
1297 auto vmbr4000
1298 iface vmbr4000 inet manual
1299 bridge_ports vxlan4000
1300 bridge_stp off
1301 bridge_fd 0
1302 hwaddress 44:39:39:FF:40:91 #must be different on each node
1303 vrf vrf1
1304 ----
1305
1306
1307 frr.conf
1308
1309 ----
1310 vrf vrf1
1311 vni 4000
1312 !
1313 router bgp 1234
1314 bgp router-id 192.168.0.2
1315 no bgp default ipv4-unicast
1316 coalesce-time 1000
1317 neighbor 192.168.0.1 remote-as 1234
1318 neighbor 192.168.0.3 remote-as 1234
1319 !
1320 address-family l2vpn evpn
1321 neighbor 192.168.0.1 activate
1322 neighbor 192.168.0.3 activate
1323 advertise-all-vni
1324 exit-address-family
1325 !
1326 router bgp 1234 vrf vrf1
1327 !
1328 bgp router-id 192.168.0.2
1329 !
1330 address-family ipv4 unicast
1331 redistribute connected
1332 exit-address-family
1333 !
1334 address-family l2vpn evpn
1335 advertise ipv4 unicast
1336 exit-address-family
1337 !
1338 line vty
1339 !
1340 ----
1341
1342
1343 * node3
1344
1345 ----
1346 auto vrf1
1347 iface vrf1
1348 vrf-table auto
1349
1350 auto eno1
1351 iface eno1 inet manual
1352
1353 auto vmbr0
1354 iface vmbr0 inet static
1355 address 192.168.0.3
1356 netmask 255.255.255.0
1357 bridge_ports eno1
1358 bridge_stp off
1359 bridge_fd 0
1360
1361 auto vxlan2
1362 iface vxlan2 inet manual
1363 vxlan-local-tunnelip 192.168.0.3
1364 bridge-learning off
1365 bridge-arp-nd-suppress on
1366 bridge-unicast-flood off
1367 bridge-multicast-flood off
1368
1369 auto vmbr2
1370 iface vmbr2 inet static
1371 bridge_ports vxlan2
1372 bridge_stp off
1373 bridge_fd 0
1374 address 10.0.2.254
1375 netmask 255.255.255.0
1376 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1377 vrf vrf1
1378
1379 auto vxlan3
1380 iface vxlan3 inet manual
1381 vxlan-local-tunnelip 192.168.0.3
1382 bridge-learning off
1383 bridge-arp-nd-suppress on
1384 bridge-unicast-flood off
1385 bridge-multicast-flood off
1386
1387 auto vmbr3
1388 iface vmbr3 inet static
1389 bridge_ports vxlan3
1390 bridge_stp off
1391 bridge_fd 0
1392 address 10.0.3.254
1393 netmask 255.255.255.0
1394 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1395 vrf vrf1
1396
1397 #interconnect vxlan-vfr l3vni
1398 auto vxlan4000
1399 iface vxlan4000 inet manual
1400 vxlan-local-tunnelip 192.168.0.3
1401 bridge-learning off
1402 bridge-arp-nd-suppress on
1403 bridge-unicast-flood off
1404 bridge-multicast-flood off
1405
1406
1407 auto vmbr4000
1408 iface vmbr4000 inet manual
1409 bridge_ports vxlan4000
1410 bridge_stp off
1411 bridge_fd 0
1412 hwaddress 44:39:39:FF:40:92 #must be different on each node
1413 vrf vrf1
1414 ----
1415
1416
1417 frr.conf
1418
1419 ----
1420 vrf vrf1
1421 vni 4000
1422 !
1423 router bgp 1234
1424 bgp router-id 192.168.0.3
1425 no bgp default ipv4-unicast
1426 coalesce-time 1000
1427 neighbor 192.168.0.1 remote-as 1234
1428 neighbor 192.168.0.2 remote-as 1234
1429 !
1430 address-family l2vpn evpn
1431 neighbor 192.168.0.1 activate
1432 neighbor 192.168.0.2 activate
1433 advertise-all-vni
1434 exit-address-family
1435 !
1436 router bgp 1234 vrf vrf1
1437 !
1438 bgp router-id 192.168.0.3
1439 !
1440 address-family ipv4 unicast
1441 redistribute connected
1442 exit-address-family
1443 !
1444 address-family l2vpn evpn
1445 advertise ipv4 unicast
1446 exit-address-family
1447 !
1448 line vty
1449 !
1450 ----
1451
1452 multiple gateway nodes
1453 ^^^^^^^^^^^^^^^^^^^^^^
1454 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1455 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1456 and announce this default gw.
1457 The external router have ecmp routes to all proxmox nodes.(balancing).
1458 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1459 vxlan the packet to final destination.
1460
1461 *node1
1462
1463 ----
1464 auto vrf1
1465 iface vrf1
1466 vrf-table auto
1467
1468 auto eno1
1469 iface eno1 inet manual
1470
1471 auto vmbr0
1472 iface vmbr0 inet static
1473 address 192.168.0.1
1474 netmask 255.255.255.0
1475 bridge_ports eno1
1476 bridge_stp off
1477 bridge_fd 0
1478
1479 auto eno2
1480 iface eno2
1481 address 172.16.0.1
1482 netmask 255.255.255.0
1483 vrf vrf1
1484 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1485 #if you have multiple external routers, you can use ecmp balancing
1486 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1487
1488 auto vxlan2
1489 iface vxlan2 inet manual
1490 vxlan-local-tunnelip 192.168.0.1
1491 bridge-learning off
1492 bridge-arp-nd-suppress on
1493 bridge-unicast-flood off
1494 bridge-multicast-flood off
1495
1496 auto vmbr2
1497 iface vmbr2 inet static
1498 bridge_ports vxlan2
1499 bridge_stp off
1500 bridge_fd 0
1501 address 10.0.2.254
1502 netmask 255.255.255.0
1503 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1504 vrf vrf1
1505
1506 auto vxlan3
1507 iface vxlan3 inet manual
1508 vxlan-local-tunnelip 192.168.0.1
1509 bridge-learning off
1510 bridge-arp-nd-suppress on
1511 bridge-unicast-flood off
1512 bridge-multicast-flood off
1513
1514 auto vmbr3
1515 iface vmbr3 inet static
1516 bridge_ports vxlan3
1517 bridge_stp off
1518 bridge_fd 0
1519 address 10.0.3.254
1520 netmask 255.255.255.0
1521 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1522 vrf vrf1
1523
1524 #interconnect vxlan-vfr l3vni
1525 auto vxlan4000
1526 iface vxlan4000 inet manual
1527 vxlan-local-tunnelip 192.168.0.1
1528 bridge-learning off
1529 bridge-arp-nd-suppress on
1530 bridge-unicast-flood off
1531 bridge-multicast-flood off
1532
1533 auto vmbr4000
1534 iface vmbr4000 inet manual
1535 bridge_ports vxlan4000
1536 bridge_stp off
1537 bridge_fd 0
1538 hwaddress 44:39:39:FF:40:90 #must be different on each node
1539 vrf vrf1
1540 ----
1541
1542
1543 frr.conf
1544
1545 ----
1546 vrf vrf1
1547 vni 4000
1548 !
1549 router bgp 1234
1550 bgp router-id 192.168.0.1
1551 no bgp default ipv4-unicast
1552 coalesce-time 1000
1553 neighbor 192.168.0.2 remote-as 1234
1554 neighbor 192.168.0.3 remote-as 1234
1555 !
1556 address-family l2vpn evpn
1557 neighbor 192.168.0.2 activate
1558 neighbor 192.168.0.3 activate
1559 advertise-all-vni
1560 exit-address-family
1561 !
1562 router bgp 1234 vrf vrf1
1563 !
1564 bgp router-id 172.16.0.1
1565 !
1566 address-family ipv4 unicast
1567 redistribute connected
1568 redistribute kernel !announce your default gw to all nodes
1569 exit-address-family
1570 !
1571 address-family l2vpn evpn
1572 advertise ipv4 unicast
1573 exit-address-family
1574 !
1575 line vty
1576 !
1577 ----
1578
1579
1580 * node2
1581
1582 ----
1583 auto vrf1
1584 iface vrf1
1585 vrf-table auto
1586
1587 auto eno1
1588 iface eno1 inet manual
1589
1590 auto vmbr0
1591 iface vmbr0 inet static
1592 address 192.168.0.2
1593 netmask 255.255.255.0
1594 bridge_ports eno1
1595 bridge_stp off
1596 bridge_fd 0
1597
1598 auto eno2
1599 iface eno2
1600 address 172.16.0.3
1601 netmask 255.255.255.0
1602 vrf vrf1
1603 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1604 #if you have multiple external routers, you can use ecmp balancing
1605 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1606
1607 auto vxlan2
1608 iface vxlan2 inet manual
1609 vxlan-local-tunnelip 192.168.0.2
1610 bridge-learning off
1611 bridge-arp-nd-suppress on
1612 bridge-unicast-flood off
1613 bridge-multicast-flood off
1614
1615 auto vmbr2
1616 iface vmbr2 inet static
1617 bridge_ports vxlan2
1618 bridge_stp off
1619 bridge_fd 0
1620 address 10.0.2.254
1621 netmask 255.255.255.0
1622 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1623 vrf vrf1
1624
1625 auto vxlan3
1626 iface vxlan3 inet manual
1627 vxlan-local-tunnelip 192.168.0.2
1628 bridge-learning off
1629 bridge-arp-nd-suppress on
1630 bridge-unicast-flood off
1631 bridge-multicast-flood off
1632
1633 auto vmbr3
1634 iface vmbr3 inet static
1635 bridge_ports vxlan3
1636 bridge_stp off
1637 bridge_fd 0
1638 address 10.0.3.254
1639 netmask 255.255.255.0
1640 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1641 vrf vrf1
1642
1643 #interconnect vxlan-vfr l3vni
1644 auto vxlan4000
1645 iface vxlan4000 inet manual
1646 vxlan-local-tunnelip 192.168.0.2
1647 bridge-learning off
1648 bridge-arp-nd-suppress on
1649 bridge-unicast-flood off
1650 bridge-multicast-flood off
1651
1652
1653 auto vmbr4000
1654 iface vmbr4000 inet manual
1655 bridge_ports vxlan4000
1656 bridge_stp off
1657 bridge_fd 0
1658 hwaddress 44:39:39:FF:40:91 #must be different on each node
1659 vrf vrf1
1660 ----
1661
1662
1663 frr.conf
1664
1665 ----
1666 vrf vrf1
1667 vni 4000
1668 !
1669 router bgp 1234
1670 bgp router-id 192.168.0.2
1671 no bgp default ipv4-unicast
1672 coalesce-time 1000
1673 neighbor 192.168.0.1 remote-as 1234
1674 neighbor 192.168.0.3 remote-as 1234
1675 !
1676 address-family l2vpn evpn
1677 neighbor 192.168.0.1 activate
1678 neighbor 192.168.0.3 activate
1679 advertise-all-vni
1680 exit-address-family
1681 !
1682 router bgp 1234 vrf vrf1
1683 !
1684 bgp router-id 172.16.0.2
1685 !
1686 address-family ipv4 unicast
1687 redistribute connected
1688 redistribute kernel !announce your default gw to all nodes
1689 exit-address-family
1690 !
1691 address-family l2vpn evpn
1692 advertise ipv4 unicast
1693 exit-address-family
1694 !
1695 line vty
1696 !
1697 ----
1698
1699
1700 * node3
1701
1702 ----
1703 auto vrf1
1704 iface vrf1
1705 vrf-table auto
1706
1707 auto eno1
1708 iface eno1 inet manual
1709
1710 auto vmbr0
1711 iface vmbr0 inet static
1712 address 192.168.0.3
1713 netmask 255.255.255.0
1714 bridge_ports eno1
1715 bridge_stp off
1716 bridge_fd 0
1717
1718 auto eno2
1719 iface eno2
1720 address 172.16.0.3
1721 netmask 255.255.255.0
1722 vrf vrf1
1723 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1724 #if you have multiple external routers, you can use ecmp balancing
1725 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1726
1727 auto vxlan2
1728 iface vxlan2 inet manual
1729 vxlan-local-tunnelip 192.168.0.3
1730 bridge-learning off
1731 bridge-arp-nd-suppress on
1732 bridge-unicast-flood off
1733 bridge-multicast-flood off
1734
1735 auto vmbr2
1736 iface vmbr2 inet static
1737 bridge_ports vxlan2
1738 bridge_stp off
1739 bridge_fd 0
1740 address 10.0.2.254
1741 netmask 255.255.255.0
1742 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1743 vrf vrf1
1744
1745 auto vxlan3
1746 iface vxlan3 inet manual
1747 vxlan-local-tunnelip 192.168.0.3
1748 bridge-learning off
1749 bridge-arp-nd-suppress on
1750 bridge-unicast-flood off
1751 bridge-multicast-flood off
1752
1753 auto vmbr3
1754 iface vmbr3 inet static
1755 bridge_ports vxlan3
1756 bridge_stp off
1757 bridge_fd 0
1758 address 10.0.3.254
1759 netmask 255.255.255.0
1760 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1761 vrf vrf1
1762
1763 #interconnect vxlan-vfr l3vni
1764 auto vxlan4000
1765 iface vxlan4000 inet manual
1766 vxlan-local-tunnelip 192.168.0.3
1767 bridge-learning off
1768 bridge-arp-nd-suppress on
1769 bridge-unicast-flood off
1770 bridge-multicast-flood off
1771
1772
1773 auto vmbr4000
1774 iface vmbr4000 inet manual
1775 bridge_ports vxlan4000
1776 bridge_stp off
1777 bridge_fd 0
1778 hwaddress 44:39:39:FF:40:92 #must be different on each node
1779 vrf vrf1
1780 ----
1781
1782
1783 frr.conf
1784
1785 ----
1786 vrf vrf1
1787 vni 4000
1788 !
1789 router bgp 1234
1790 bgp router-id 192.168.0.3
1791 no bgp default ipv4-unicast
1792 coalesce-time 1000
1793 neighbor 192.168.0.1 remote-as 1234
1794 neighbor 192.168.0.2 remote-as 1234
1795 !
1796 address-family l2vpn evpn
1797 neighbor 192.168.0.1 activate
1798 neighbor 192.168.0.2 activate
1799 advertise-all-vni
1800 exit-address-family
1801 !
1802 router bgp 1234 vrf vrf1
1803 !
1804 bgp router-id 172.16.0.3
1805 !
1806 address-family ipv4 unicast
1807 redistribute connected
1808 redistribute kernel !announce your default gw to all nodes
1809 exit-address-family
1810 !
1811 address-family l2vpn evpn
1812 advertise ipv4 unicast
1813 exit-address-family
1814 !
1815 line vty
1816 !
1817 ----
1818
1819 Note
1820 ^^^^
1821
1822 If your external router don't support ecmp to reach multiple proxmox nodes,
1823 you can setup an HA floating vip on proxmox nodes with vrrp
1824
1825 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1826 Node1 is the primary and failover to node2 in case of failure.
1827
1828
1829 * node1
1830
1831 ----
1832 auto eno2
1833 iface eno2
1834 address 172.16.0.1
1835 netmask 255.255.255.0
1836 vrf vrf1
1837 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1838 vrrp-id 1
1839 vrrp-priority 1
1840 vrrp-virtual-ip 172.16.0.10
1841 ----
1842
1843 * node2
1844
1845 ----
1846 auto eno2
1847 iface eno2
1848 address 172.16.0.2
1849 netmask 255.255.255.0
1850 vrf vrf1
1851 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1852 vrrp-id 1
1853 vrrp-priority 2
1854 vrrp-virtual-ip 172.16.0.10
1855 ----
1856
1857