vxlan : add missing vxlan-id
[pve-docs.git] / vxlan-and-evpn.adoc
1
2 ////
3
4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
7
8 ////
9
10
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
13
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
18
19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
21
22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
24
25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
26
27 multicast mode
28 ^^^^^^^^^^^^^^
29
30 This scenario relies in head end replication, meaning that end host in case
31 of not having any entry for the destination MAC address will send out an ARP
32 to other devices / VTEPs in the VXLAN network.
33 This is done by sending the request to the VXLAN multicast group,
34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
35
36
37 * node1
38
39 ----
40 auto eno1
41 iface eno1 inet manual
42 mtu 1550
43
44 auto vmbr0
45 iface vmbr0 inet static
46 address 192.168.0.1
47 netmask 255.255.255.0
48 bridge_ports eno1
49 bridge_stp off
50 bridge_fd 0
51
52 auto vxlan2
53 iface vxlan2 inet manual
54 vxlan-id 2
55 vxlan-svcnodeip 225.20.1.1
56 vxlan-physdev eno1
57
58 auto vmbr2
59 iface vmbr2 inet manual
60 bridge_ports vxlan2
61 bridge_stp off
62 bridge_fd 0
63
64 auto vxlan3
65 iface vxlan3 inet manual
66 vxlan-id 3
67 vxlan-svcnodeip 225.20.1.1
68 vxlan-physdev eno1
69
70 auto vmbr3
71 iface vmbr3 inet manual
72 bridge_ports vxlan3
73 bridge_stp off
74 bridge_fd 0
75 ----
76
77
78 * node2
79
80 ----
81 auto eno1
82 iface eno1 inet manual
83 mtu 1550
84
85 auto vmbr0
86 iface vmbr0 inet static
87 address 192.168.0.2
88 netmask 255.255.255.0
89 bridge_ports eno1
90 bridge_stp off
91 bridge_fd 0
92
93 auto vxlan2
94 iface vxlan2 inet manual
95 vxlan-id 2
96 vxlan-svcnodeip 225.20.1.1
97 vxlan-physdev eno1
98
99 auto vmbr2
100 iface vmbr2 inet manual
101 bridge_ports vxlan2
102 bridge_stp off
103 bridge_fd 0
104
105
106 auto vxlan3
107 iface vxlan3 inet manual
108 vxlan-id 3
109 vxlan-svcnodeip 225.20.1.1
110 vxlan-physdev eno1
111
112 auto vmbr3
113 iface vmbr3 inet manual
114 bridge_ports vxlan3
115 bridge_stp off
116 bridge_fd 0
117 ----
118
119
120 * node3
121
122 ----
123 auto eno1
124 iface eno1 inet manual
125 mtu 1550
126
127 auto vmbr0
128 iface vmbr0 inet static
129 address 192.168.0.3
130 netmask 255.255.255.0
131 bridge_ports eno1
132 bridge_stp off
133 bridge_fd 0
134
135 auto vxlan2
136 iface vxlan2 inet manual
137 vxlan-id 2
138 vxlan-svcnodeip 225.20.1.1
139 vxlan-physdev eno1
140
141 auto vmbr2
142 iface vmbr2 inet manual
143 bridge_ports vxlan2
144 bridge_stp off
145 bridge_fd 0
146
147
148 auto vxlan3
149 iface vxlan3 inet manual
150 vxlan-id 3
151 vxlan-svcnodeip 225.20.1.1
152 vxlan-physdev eno1
153
154 auto vmbr3
155 iface vmbr3 inet manual
156 bridge_ports vxlan3
157 bridge_stp off
158 bridge_fd 0
159 ----
160
161
162 unicast mode
163 ^^^^^^^^^^^^
164
165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
166 The VXLAN is defined without a remote multicast group.
167 Instead, all the remote VTEPs are associated with the all-zero address:
168 a BUM frame will be duplicated to all these destinations.
169 The VXLAN device will still learn remote addresses automatically using source-address learning.
170
171 * node1
172
173 ----
174 auto eno1
175 iface eno1 inet manual
176 mtu 1550
177
178 auto vmbr0
179 iface vmbr0 inet static
180 address 192.168.0.1
181 netmask 255.255.255.0
182 bridge_ports eno1
183 bridge_stp off
184 bridge_fd 0
185
186
187 auto vxlan2
188 iface vxlan2 inet manual
189 vxlan-id 2
190 vxlan_remoteip 192.168.0.2
191 vxlan_remoteip 192.168.0.3
192
193
194 auto vmbr2
195 iface vmbr2 inet manual
196 bridge_ports vxlan2
197 bridge_stp off
198 bridge_fd 0
199
200
201 auto vxlan3
202 iface vxlan2 inet manual
203 vxlan-id 3
204 vxlan_remoteip 192.168.0.2
205 vxlan_remoteip 192.168.0.3
206
207
208 auto vmbr3
209 iface vmbr3 inet manual
210 bridge_ports vxlan3
211 bridge_stp off
212 bridge_fd 0
213 ----
214
215
216 * node2
217
218 ----
219 auto eno1
220 iface eno1 inet manual
221 mtu 1550
222
223 auto vmbr0
224 iface vmbr0 inet static
225 address 192.168.0.2
226 netmask 255.255.255.0
227 bridge_ports eno1
228 bridge_stp off
229 bridge_fd 0
230
231 auto vxlan2
232 iface vxlan2 inet manual
233 vxlan-id 2
234 vxlan_remoteip 192.168.0.1
235 vxlan_remoteip 192.168.0.3
236
237
238
239 auto vmbr2
240 iface vmbr2 inet manual
241 bridge_ports vxlan2
242 bridge_stp off
243 bridge_fd 0
244
245 auto vxlan3
246 iface vxlan2 inet manual
247 vxlan-id 3
248 vxlan_remoteip 192.168.0.1
249 vxlan_remoteip 192.168.0.3
250
251
252 auto vmbr3
253 iface vmbr3 inet manual
254 bridge_ports vxlan3
255 bridge_stp off
256 bridge_fd 0
257 ----
258
259
260 * node3
261
262 ----
263 auto eno1
264 iface eno1 inet manual
265 mtu 1550
266
267 auto vmbr0
268 iface vmbr0 inet static
269 address 192.168.0.3
270 netmask 255.255.255.0
271 bridge_ports eno1
272 bridge_stp off
273 bridge_fd 0
274
275 auto vxlan2
276 iface vxlan2 inet manual
277 vxlan-id 2
278 vxlan_remoteip 192.168.0.2
279 vxlan_remoteip 192.168.0.3
280
281
282
283 auto vmbr2
284 iface vmbr2 inet manual
285 bridge_ports vxlan2
286 bridge_stp off
287 bridge_fd 0
288
289 auto vxlan3
290 iface vxlan2 inet manual
291 vxlan-id 3
292 vxlan_remoteip 192.168.0.2
293 vxlan_remoteip 192.168.0.3
294
295
296 auto vmbr3
297 iface vmbr3 inet manual
298 bridge_ports vxlan3
299 bridge_stp off
300 bridge_fd 0
301 ----
302
303
304 bgp-evpn
305 ^^^^^^^^
306
307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
309
310 The control plane used here is FRR, a bgp routing software.
311 Each node in the proxmox cluster peer with each others nodes.
312 For bigger networks, or multiple proxmox clusters,
313 it's possible to use external bgp route reflector servers.
314
315 * node1
316
317 ----
318 auto eno1
319 iface eno1 inet manual
320 mtu 1550
321
322 auto vmbr0
323 iface vmbr0 inet static
324 address 192.168.0.1
325 netmask 255.255.255.0
326 bridge_ports eno1
327 bridge_stp off
328 bridge_fd 0
329
330 auto vxlan2
331 iface vxlan2 inet manual
332 vxlan-id 2
333 vxlan-local-tunnelip 192.168.0.1
334 bridge-learning off
335 bridge-arp-nd-suppress on
336 bridge-unicast-flood off
337 bridge-multicast-flood off
338
339
340 auto vmbr2
341 iface vmbr2 inet manual
342 bridge_ports vxlan2
343 bridge_stp off
344 bridge_fd 0
345
346
347 auto vxlan3
348 iface vxlan3 inet manual
349 vxlan-id 3
350 vxlan-local-tunnelip 192.168.0.1
351 bridge-learning off
352 bridge-arp-nd-suppress on
353 bridge-unicast-flood off
354 bridge-multicast-flood off
355
356
357 auto vmbr3
358 iface vmbr3 inet manual
359 bridge_ports vxlan3
360 bridge_stp off
361 bridge_fd 0
362 ----
363
364
365 /etc/frr/frr.conf
366
367 ----
368 router bgp 1234
369 no bgp default ipv4-unicast
370 coalesce-time 1000
371 neighbor 192.168.0.2 remote-as 1234
372 neighbor 192.168.0.3 remote-as 1234
373 !
374 address-family l2vpn evpn
375 neighbor 192.168.0.2 activate
376 neighbor 192.168.0.3 activate
377 advertise-all-vni
378 exit-address-family
379 !
380 line vty
381 !
382 ----
383
384
385 * node2
386
387 ----
388 auto eno1
389 iface eno1 inet manual
390 mtu 1550
391
392 auto vmbr0
393 iface vmbr0 inet static
394 address 192.168.0.2
395 netmask 255.255.255.0
396 bridge_ports eno1
397 bridge_stp off
398 bridge_fd 0
399
400 auto vxlan2
401 iface vxlan2 inet manual
402 vxlan-id 2
403 vxlan-local-tunnelip 192.168.0.2
404 bridge-learning off
405 bridge-arp-nd-suppress on
406 bridge-unicast-flood off
407 bridge-multicast-flood off
408
409
410 auto vmbr2
411 iface vmbr2 inet manual
412 bridge_ports vxlan2
413 bridge_stp off
414 bridge_fd 0
415
416 auto vxlan3
417 iface vxlan3 inet manual
418 vxlan-id 3
419 vxlan-local-tunnelip 192.168.0.2
420 bridge-learning off
421 bridge-arp-nd-suppress on
422 bridge-unicast-flood off
423 bridge-multicast-flood off
424
425
426 auto vmbr3
427 iface vmbr3 inet manual
428 bridge_ports vxlan3
429 bridge_stp off
430 bridge_fd 0
431 ----
432
433
434 /etc/frr/frr.conf
435
436 ----
437 router bgp 1234
438 no bgp default ipv4-unicast
439 coalesce-time 1000
440 neighbor 192.168.0.1 remote-as 1234
441 neighbor 192.168.0.3 remote-as 1234
442 !
443 address-family l2vpn evpn
444 neighbor 192.168.0.1 activate
445 neighbor 192.168.0.3 activate
446 advertise-all-vni
447 exit-address-family
448 !
449 line vty
450 !
451 ----
452
453
454 * node3
455
456 ----
457 auto eno1
458 iface eno1 inet manual
459 mtu 1550
460
461 auto vmbr0
462 iface vmbr0 inet static
463 address 192.168.0.2
464 netmask 255.255.255.0
465 bridge_ports eno1
466 bridge_stp off
467 bridge_fd 0
468
469 auto vxlan2
470 iface vxlan2 inet manual
471 vxlan-id 2
472 vxlan-local-tunnelip 192.168.0.3
473 bridge-learning off
474 bridge-arp-nd-suppress on
475 bridge-unicast-flood off
476 bridge-multicast-flood off
477
478
479 auto vmbr2
480 iface vmbr2 inet manual
481 bridge_ports vxlan2
482 bridge_stp off
483 bridge_fd 0
484
485 auto vxlan3
486 iface vxlan3 inet manual
487 vxlan-id 3
488 vxlan-local-tunnelip 192.168.0.3
489 bridge-learning off
490 bridge-arp-nd-suppress on
491 bridge-unicast-flood off
492 bridge-multicast-flood off
493
494
495 auto vmbr3
496 iface vmbr3 inet manual
497 bridge_ports vxlan3
498 bridge_stp off
499 bridge_fd 0
500 ----
501
502
503 /etc/frr/frr.conf
504
505
506 ----
507 router bgp 1234
508 no bgp default ipv4-unicast
509 coalesce-time 1000
510 neighbor 192.168.0.1 remote-as 1234
511 neighbor 192.168.0.2 remote-as 1234
512 !
513 address-family l2vpn evpn
514 neighbor 192.168.0.1 activate
515 neighbor 192.168.0.2 activate
516 advertise-all-vni
517 exit-address-family
518 !
519 line vty
520 !
521 ----
522
523 VXLAN layer3 routing with anycast gateway
524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
525
526 With this need, each vmbr bridge will be the gateway for the vm.
527 Same vmbr on different node, will have same ip address and same mac address,
528 to have working vm live migration and no network disruption.
529
530 VXLAN layer3 routing only work with FRR and non-aware bridge.
531 (vlan aware bridge support is buggy currently).
532
533 asymmetric model
534 ^^^^^^^^^^^^^^^^
535
536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
537
538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
539 but only bridging on the egress.
540 This results in bi-directional VXLAN traffic traveling on different VNIs
541 in each direction (always the destination VNI) across the routed infrastructure.
542
543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
544
545
546 sysctl.conf tuning
547
548 ----
549 #enable routing
550 net.ipv4.ip_forward=1
551 net.ipv6.conf.all.forwarding=1
552 ----
553
554 * node1
555
556 ----
557 auto eno1
558 iface eno1 inet manual
559 mtu 1550
560
561 auto vmbr0
562 iface vmbr0 inet static
563 address 192.168.0.1
564 netmask 255.255.255.0
565 bridge_ports eno1
566 bridge_stp off
567 bridge_fd 0
568
569 auto vxlan2
570 iface vxlan2 inet manual
571 vxlan-id 2
572 vxlan-local-tunnelip 192.168.0.1
573 bridge-learning off
574 bridge-arp-nd-suppress on
575 bridge-unicast-flood off
576 bridge-multicast-flood off
577
578
579 auto vmbr2
580 iface vmbr2 inet static
581 address 10.0.2.254
582 netmask 255.255.255.0
583 hwaddress 44:39:39:FF:40:94
584 bridge_ports vxlan2
585 bridge_stp off
586 bridge_fd 0
587
588
589 auto vxlan3
590 iface vxlan3 inet manual
591 vxlan-id 3
592 vxlan-local-tunnelip 192.168.0.1
593 bridge-learning off
594 bridge-arp-nd-suppress on
595 bridge-unicast-flood off
596 bridge-multicast-flood off
597
598
599 auto vmbr3
600 iface vmbr3 inet static
601 address 10.0.3.254
602 netmask 255.255.255.0
603 hwaddress 44:39:39:FF:40:94
604 bridge_ports vxlan3
605 bridge_stp off
606 bridge_fd 0
607 ----
608
609
610 frr.conf
611
612 ----
613 router bgp 1234
614 bgp router-id 192.168.0.1
615 no bgp default ipv4-unicast
616 coalesce-time 1000
617 neighbor 192.168.0.2 remote-as 1234
618 neighbor 192.168.0.3 remote-as 1234
619 !
620 address-family l2vpn evpn
621 neighbor 192.168.0.2 activate
622 neighbor 192.168.0.3 activate
623 advertise-all-vni
624 exit-address-family
625 !
626 line vty
627 !
628 ----
629
630
631 * node2
632
633 ----
634 auto eno1
635 iface eno1 inet manual
636 mtu 1550
637
638 auto vmbr0
639 iface vmbr0 inet static
640 address 192.168.0.2
641 netmask 255.255.255.0
642 bridge_ports eno1
643 bridge_stp off
644 bridge_fd 0
645
646 auto vxlan2
647 iface vxlan2 inet manual
648 vxlan-id 2
649 vxlan-local-tunnelip 192.168.0.2
650 bridge-learning off
651 bridge-arp-nd-suppress on
652 bridge-unicast-flood off
653 bridge-multicast-flood off
654
655
656 auto vmbr2
657 iface vmbr2 inet static
658 address 10.0.2.254
659 netmask 255.255.255.0
660 hwaddress 44:39:39:FF:40:94
661 bridge_ports vxlan2
662 bridge_stp off
663 bridge_fd 0
664
665
666 auto vxlan3
667 iface vxlan3 inet manual
668 vxlan-id 3
669 vxlan-local-tunnelip 192.168.0.2
670 bridge-learning off
671 bridge-arp-nd-suppress on
672 bridge-unicast-flood off
673 bridge-multicast-flood off
674
675
676 auto vmbr3
677 iface vmbr3 inet static
678 address 10.0.3.254
679 netmask 255.255.255.0
680 hwaddress 44:39:39:FF:40:94
681 bridge_ports vxlan3
682 bridge_stp off
683 bridge_fd 0
684 ----
685
686
687 frr.conf
688
689 ----
690 router bgp 1234
691 bgp router-id 192.168.0.2
692 no bgp default ipv4-unicast
693 coalesce-time 1000
694 neighbor 192.168.0.1 remote-as 1234
695 neighbor 192.168.0.3 remote-as 1234
696 !
697 address-family l2vpn evpn
698 neighbor 192.168.0.1 activate
699 neighbor 192.168.0.3 activate
700 advertise-all-vni
701 exit-address-family
702 !
703 line vty
704 !
705 ----
706
707
708 * node3
709
710 ----
711 auto eno1
712 iface eno1 inet manual
713 mtu 1550
714
715 auto vmbr0
716 iface vmbr0 inet static
717 address 192.168.0.3
718 netmask 255.255.255.0
719 bridge_ports eno1
720 bridge_stp off
721 bridge_fd 0
722
723 auto vxlan2
724 iface vxlan2 inet manual
725 vxlan-id 2
726 vxlan-local-tunnelip 192.168.0.3
727 bridge-learning off
728 bridge-arp-nd-suppress on
729 bridge-unicast-flood off
730 bridge-multicast-flood off
731
732
733 auto vmbr2
734 iface vmbr2 inet static
735 address 10.0.2.254
736 netmask 255.255.255.0
737 hwaddress 44:39:39:FF:40:94
738 bridge_ports vxlan2
739 bridge_stp off
740 bridge_fd 0
741
742
743 auto vxlan3
744 iface vxlan3 inet manual
745 vxlan-id 3
746 vxlan-local-tunnelip 192.168.0.3
747 bridge-learning off
748 bridge-arp-nd-suppress on
749 bridge-unicast-flood off
750 bridge-multicast-flood off
751
752
753 auto vmbr3
754 iface vmbr3 inet static
755 address 10.0.3.254
756 netmask 255.255.255.0
757 hwaddress 44:39:39:FF:40:94
758 bridge_ports vxlan3
759 bridge_stp off
760 bridge_fd 0
761 ----
762
763
764 frr.conf
765
766 ----
767 router bgp 1234
768 bgp router-id 192.168.0.3
769 no bgp default ipv4-unicast
770 coalesce-time 1000
771 neighbor 192.168.0.1 remote-as 1234
772 neighbor 192.168.0.2 remote-as 1234
773 !
774 address-family l2vpn evpn
775 neighbor 192.168.0.1 activate
776 neighbor 192.168.0.2 activate
777 advertise-all-vni
778 exit-address-family
779 !
780 line vty
781 !
782 ----
783
784
785 symmetric model
786 ^^^^^^^^^^^^^^^
787
788 With this model, you don't need to have all vxlan on all nodes.
789 This model will also be needed to route traffic to an external router.
790
791 The symmetric model routes and bridges on both the ingress and the egress leafs.
792 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
793 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
794 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
795 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
796
797 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
798
799 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
800
801 sysctl.conf tuning
802
803 ----
804 #enable routing
805 net.ipv4.ip_forward=1
806 net.ipv6.conf.all.forwarding=1
807 #disable reverse path filtering
808 net.ipv4.conf.default.rp_filter=0
809 net.ipv4.conf.all.rp_filter=0
810 ----
811
812 * node1
813
814 ----
815 auto vrf1
816 iface vrf1
817 vrf-table auto
818
819 auto eno1
820 iface eno1 inet manual
821 mtu 1550
822
823 auto vmbr0
824 iface vmbr0 inet static
825 address 192.168.0.1
826 netmask 255.255.255.0
827 bridge_ports eno1
828 bridge_stp off
829 bridge_fd 0
830
831 auto vxlan2
832 iface vxlan2 inet manual
833 vxlan-id 2
834 vxlan-local-tunnelip 192.168.0.1
835 bridge-learning off
836 bridge-arp-nd-suppress on
837 bridge-unicast-flood off
838 bridge-multicast-flood off
839
840 auto vmbr2
841 iface vmbr2 inet static
842 bridge_ports vxlan2
843 bridge_stp off
844 bridge_fd 0
845 address 10.0.2.254
846 netmask 255.255.255.0
847 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
848 vrf vrf1
849
850 auto vxlan3
851 iface vxlan3 inet manual
852 vxlan-id 3
853 vxlan-local-tunnelip 192.168.0.1
854 bridge-learning off
855 bridge-arp-nd-suppress on
856 bridge-unicast-flood off
857 bridge-multicast-flood off
858
859 auto vmbr3
860 iface vmbr3 inet static
861 bridge_ports vxlan3
862 bridge_stp off
863 bridge_fd 0
864 address 10.0.3.254
865 netmask 255.255.255.0
866 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
867 vrf vrf1
868
869 #interconnect vxlan-vfr l3vni
870 auto vxlan4000
871 iface vxlan4000 inet manual
872 vxlan-id 4000
873 vxlan-local-tunnelip 192.168.0.1
874 bridge-learning off
875 bridge-arp-nd-suppress on
876 bridge-unicast-flood off
877 bridge-multicast-flood off
878
879
880 auto vmbr4000
881 iface vmbr4000 inet manual
882 bridge_ports vxlan4000
883 bridge_stp off
884 bridge_fd 0
885 hwaddress 44:39:39:FF:40:90 #must be different on each node
886 vrf vrf1
887 ----
888
889 frr.conf
890
891 ----
892 vrf vrf1
893 vni 4000
894 !
895 router bgp 1234
896 bgp router-id 192.168.0.1
897 no bgp default ipv4-unicast
898 coalesce-time 1000
899 neighbor 192.168.0.2 remote-as 1234
900 neighbor 192.168.0.3 remote-as 1234
901 !
902 address-family l2vpn evpn
903 neighbor 192.168.0.2 activate
904 neighbor 192.168.0.3 activate
905 advertise-all-vni
906 exit-address-family
907 !
908 router bgp 1234 vrf vrf1
909 !
910 bgp router-id 192.168.0.1
911 !
912 address-family ipv4 unicast
913 redistribute connected
914 exit-address-family
915 !
916 address-family l2vpn evpn
917 advertise ipv4 unicast
918 exit-address-family
919 !
920 line vty
921 !
922 ----
923
924
925 * node2
926
927 ----
928 auto vrf1
929 iface vrf1
930 vrf-table auto
931
932 auto eno1
933 iface eno1 inet manual
934 mtu 1550
935
936 auto vmbr0
937 iface vmbr0 inet static
938 address 192.168.0.2
939 netmask 255.255.255.0
940 bridge_ports eno1
941 bridge_stp off
942 bridge_fd 0
943
944 auto vxlan2
945 iface vxlan2 inet manual
946 vxlan-id 2
947 vxlan-local-tunnelip 192.168.0.2
948 bridge-learning off
949 bridge-arp-nd-suppress on
950 bridge-unicast-flood off
951 bridge-multicast-flood off
952
953 auto vmbr2
954 iface vmbr2 inet static
955 bridge_ports vxlan2
956 bridge_stp off
957 bridge_fd 0
958 address 10.0.2.254
959 netmask 255.255.255.0
960 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
961 vrf vrf1
962
963 auto vxlan3
964 iface vxlan3 inet manual
965 vxlan-id 3
966 vxlan-local-tunnelip 192.168.0.2
967 bridge-learning off
968 bridge-arp-nd-suppress on
969 bridge-unicast-flood off
970 bridge-multicast-flood off
971
972 auto vmbr3
973 iface vmbr3 inet static
974 bridge_ports vxlan3
975 bridge_stp off
976 bridge_fd 0
977 address 10.0.3.254
978 netmask 255.255.255.0
979 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
980 vrf vrf1
981
982 #interconnect vxlan-vfr l3vni
983 auto vxlan4000
984 iface vxlan4000 inet manual
985 vxlan-id 4000
986 vxlan-local-tunnelip 192.168.0.2
987 bridge-learning off
988 bridge-arp-nd-suppress on
989 bridge-unicast-flood off
990 bridge-multicast-flood off
991
992
993 auto vmbr4000
994 iface vmbr4000 inet manual
995 bridge_ports vxlan4000
996 bridge_stp off
997 bridge_fd 0
998 hwaddress 44:39:39:FF:40:91 #must be different on each node
999 vrf vrf1
1000 ----
1001
1002
1003 frr.conf
1004
1005 ----
1006 vrf vrf1
1007 vni 4000
1008 !
1009 router bgp 1234
1010 bgp router-id 192.168.0.2
1011 no bgp default ipv4-unicast
1012 coalesce-time 1000
1013 neighbor 192.168.0.1 remote-as 1234
1014 neighbor 192.168.0.3 remote-as 1234
1015 !
1016 address-family l2vpn evpn
1017 neighbor 192.168.0.1 activate
1018 neighbor 192.168.0.3 activate
1019 advertise-all-vni
1020 exit-address-family
1021 !
1022 router bgp 1234 vrf vrf1
1023 !
1024 bgp router-id 192.168.0.2
1025 !
1026 address-family ipv4 unicast
1027 redistribute connected
1028 exit-address-family
1029 !
1030 address-family l2vpn evpn
1031 advertise ipv4 unicast
1032 exit-address-family
1033 !
1034 line vty
1035 !
1036 ----
1037
1038
1039 * node3
1040
1041 ----
1042 auto vrf1
1043 iface vrf1
1044 vrf-table auto
1045
1046 auto eno1
1047 iface eno1 inet manual
1048 mtu 1550
1049
1050 auto vmbr0
1051 iface vmbr0 inet static
1052 address 192.168.0.3
1053 netmask 255.255.255.0
1054 bridge_ports eno1
1055 bridge_stp off
1056 bridge_fd 0
1057
1058 auto vxlan2
1059 iface vxlan2 inet manual
1060 vxlan-id 2
1061 vxlan-local-tunnelip 192.168.0.3
1062 bridge-learning off
1063 bridge-arp-nd-suppress on
1064 bridge-unicast-flood off
1065 bridge-multicast-flood off
1066
1067 auto vmbr2
1068 iface vmbr2 inet static
1069 bridge_ports vxlan2
1070 bridge_stp off
1071 bridge_fd 0
1072 address 10.0.2.254
1073 netmask 255.255.255.0
1074 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1075 vrf vrf1
1076
1077 auto vxlan3
1078 iface vxlan3 inet manual
1079 vxlan-id 3
1080 vxlan-local-tunnelip 192.168.0.3
1081 bridge-learning off
1082 bridge-arp-nd-suppress on
1083 bridge-unicast-flood off
1084 bridge-multicast-flood off
1085
1086 auto vmbr3
1087 iface vmbr3 inet static
1088 bridge_ports vxlan3
1089 bridge_stp off
1090 bridge_fd 0
1091 address 10.0.3.254
1092 netmask 255.255.255.0
1093 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1094 vrf vrf1
1095
1096 #interconnect vxlan-vfr l3vni
1097 auto vxlan4000
1098 iface vxlan4000 inet manual
1099 vxlan-id 4000
1100 vxlan-local-tunnelip 192.168.0.3
1101 bridge-learning off
1102 bridge-arp-nd-suppress on
1103 bridge-unicast-flood off
1104 bridge-multicast-flood off
1105
1106
1107 auto vmbr4000
1108 iface vmbr4000 inet manual
1109 bridge_ports vxlan4000
1110 bridge_stp off
1111 bridge_fd 0
1112 hwaddress 44:39:39:FF:40:92 #must be different on each node
1113 vrf vrf1
1114 ----
1115
1116
1117 frr.conf
1118
1119 ----
1120 vrf vrf1
1121 vni 4000
1122 !
1123 router bgp 1234
1124 bgp router-id 192.168.0.3
1125 no bgp default ipv4-unicast
1126 coalesce-time 1000
1127 neighbor 192.168.0.1 remote-as 1234
1128 neighbor 192.168.0.2 remote-as 1234
1129 !
1130 address-family l2vpn evpn
1131 neighbor 192.168.0.1 activate
1132 neighbor 192.168.0.2 activate
1133 advertise-all-vni
1134 exit-address-family
1135 !
1136 router bgp 1234 vrf vrf1
1137 !
1138 bgp router-id 192.168.0.3
1139 !
1140 address-family ipv4 unicast
1141 redistribute connected
1142 exit-address-family
1143 !
1144 address-family l2vpn evpn
1145 advertise ipv4 unicast
1146 exit-address-family
1147 !
1148 line vty
1149 !
1150 ----
1151
1152 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1153 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1154 Routing to outside need the symmetric model.
1155
1156 1 gateway node
1157 ^^^^^^^^^^^^^^
1158 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1159 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1160 and announce this default gw to other proxmox nodes.
1161
1162
1163 *node1
1164
1165 ----
1166 auto vrf1
1167 iface vrf1
1168 vrf-table auto
1169
1170 auto eno1
1171 iface eno1 inet manual
1172 mtu 1550
1173
1174 auto vmbr0
1175 iface vmbr0 inet static
1176 address 192.168.0.1
1177 netmask 255.255.255.0
1178 bridge_ports eno1
1179 bridge_stp off
1180 bridge_fd 0
1181
1182 auto eno2
1183 iface eno2
1184 address 172.16.0.1
1185 netmask 255.255.255.0
1186 vrf vrf1
1187 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1188 #if you have multiple external routers, you can use ecmp balancing
1189 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1190
1191 auto vxlan2
1192 iface vxlan2 inet manual
1193 vxlan-id 2
1194 vxlan-local-tunnelip 192.168.0.1
1195 bridge-learning off
1196 bridge-arp-nd-suppress on
1197 bridge-unicast-flood off
1198 bridge-multicast-flood off
1199
1200 auto vmbr2
1201 iface vmbr2 inet static
1202 bridge_ports vxlan2
1203 bridge_stp off
1204 bridge_fd 0
1205 address 10.0.2.254
1206 netmask 255.255.255.0
1207 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1208 vrf vrf1
1209
1210 auto vxlan3
1211 iface vxlan3 inet manual
1212 vxlan-id 3
1213 vxlan-local-tunnelip 192.168.0.1
1214 bridge-learning off
1215 bridge-arp-nd-suppress on
1216 bridge-unicast-flood off
1217 bridge-multicast-flood off
1218
1219 auto vmbr3
1220 iface vmbr3 inet static
1221 bridge_ports vxlan3
1222 bridge_stp off
1223 bridge_fd 0
1224 address 10.0.3.254
1225 netmask 255.255.255.0
1226 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1227 vrf vrf1
1228
1229 #interconnect vxlan-vfr l3vni
1230 auto vxlan4000
1231 iface vxlan4000 inet manual
1232 vxlan-id 4000
1233 vxlan-local-tunnelip 192.168.0.1
1234 bridge-learning off
1235 bridge-arp-nd-suppress on
1236 bridge-unicast-flood off
1237 bridge-multicast-flood off
1238
1239 auto vmbr4000
1240 iface vmbr4000 inet manual
1241 bridge_ports vxlan4000
1242 bridge_stp off
1243 bridge_fd 0
1244 hwaddress 44:39:39:FF:40:90 #must be different on each node
1245 vrf vrf1
1246 ----
1247
1248
1249 frr.conf
1250
1251 ----
1252 vrf vrf1
1253 vni 4000
1254 !
1255 router bgp 1234
1256 bgp router-id 192.168.0.1
1257 no bgp default ipv4-unicast
1258 coalesce-time 1000
1259 neighbor 192.168.0.2 remote-as 1234
1260 neighbor 192.168.0.3 remote-as 1234
1261 !
1262 address-family l2vpn evpn
1263 neighbor 192.168.0.2 activate
1264 neighbor 192.168.0.3 activate
1265 advertise-all-vni
1266 exit-address-family
1267 !
1268 router bgp 1234 vrf vrf1
1269 !
1270 bgp router-id 172.16.0.1
1271 !
1272 address-family ipv4 unicast
1273 redistribute connected
1274 redistribute kernel !announce your default gw to all nodes
1275 exit-address-family
1276 !
1277 address-family l2vpn evpn
1278 advertise ipv4 unicast
1279 exit-address-family
1280 !
1281 line vty
1282 !
1283 ----
1284
1285
1286 * node2
1287
1288 ----
1289 auto vrf1
1290 iface vrf1
1291 vrf-table auto
1292
1293 auto eno1
1294 iface eno1 inet manual
1295 mtu 1550
1296
1297 auto vmbr0
1298 iface vmbr0 inet static
1299 address 192.168.0.2
1300 netmask 255.255.255.0
1301 bridge_ports eno1
1302 bridge_stp off
1303 bridge_fd 0
1304
1305 auto vxlan2
1306 iface vxlan2 inet manual
1307 vxlan-id 2
1308 vxlan-local-tunnelip 192.168.0.2
1309 bridge-learning off
1310 bridge-arp-nd-suppress on
1311 bridge-unicast-flood off
1312 bridge-multicast-flood off
1313
1314 auto vmbr2
1315 iface vmbr2 inet static
1316 bridge_ports vxlan2
1317 bridge_stp off
1318 bridge_fd 0
1319 address 10.0.2.254
1320 netmask 255.255.255.0
1321 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1322 vrf vrf1
1323
1324 auto vxlan3
1325 iface vxlan3 inet manual
1326 vxlan-id 3
1327 vxlan-local-tunnelip 192.168.0.2
1328 bridge-learning off
1329 bridge-arp-nd-suppress on
1330 bridge-unicast-flood off
1331 bridge-multicast-flood off
1332
1333 auto vmbr3
1334 iface vmbr3 inet static
1335 bridge_ports vxlan3
1336 bridge_stp off
1337 bridge_fd 0
1338 address 10.0.3.254
1339 netmask 255.255.255.0
1340 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1341 vrf vrf1
1342
1343 #interconnect vxlan-vfr l3vni
1344 auto vxlan4000
1345 iface vxlan4000 inet manual
1346 vxlan-id 4000
1347 vxlan-local-tunnelip 192.168.0.2
1348 bridge-learning off
1349 bridge-arp-nd-suppress on
1350 bridge-unicast-flood off
1351 bridge-multicast-flood off
1352
1353
1354 auto vmbr4000
1355 iface vmbr4000 inet manual
1356 bridge_ports vxlan4000
1357 bridge_stp off
1358 bridge_fd 0
1359 hwaddress 44:39:39:FF:40:91 #must be different on each node
1360 vrf vrf1
1361 ----
1362
1363
1364 frr.conf
1365
1366 ----
1367 vrf vrf1
1368 vni 4000
1369 !
1370 router bgp 1234
1371 bgp router-id 192.168.0.2
1372 no bgp default ipv4-unicast
1373 coalesce-time 1000
1374 neighbor 192.168.0.1 remote-as 1234
1375 neighbor 192.168.0.3 remote-as 1234
1376 !
1377 address-family l2vpn evpn
1378 neighbor 192.168.0.1 activate
1379 neighbor 192.168.0.3 activate
1380 advertise-all-vni
1381 exit-address-family
1382 !
1383 router bgp 1234 vrf vrf1
1384 !
1385 bgp router-id 192.168.0.2
1386 !
1387 address-family ipv4 unicast
1388 redistribute connected
1389 exit-address-family
1390 !
1391 address-family l2vpn evpn
1392 advertise ipv4 unicast
1393 exit-address-family
1394 !
1395 line vty
1396 !
1397 ----
1398
1399
1400 * node3
1401
1402 ----
1403 auto vrf1
1404 iface vrf1
1405 vrf-table auto
1406
1407 auto eno1
1408 iface eno1 inet manual
1409 mtu 1550
1410
1411 auto vmbr0
1412 iface vmbr0 inet static
1413 address 192.168.0.3
1414 netmask 255.255.255.0
1415 bridge_ports eno1
1416 bridge_stp off
1417 bridge_fd 0
1418
1419 auto vxlan2
1420 iface vxlan2 inet manual
1421 vxlan-id 2
1422 vxlan-local-tunnelip 192.168.0.3
1423 bridge-learning off
1424 bridge-arp-nd-suppress on
1425 bridge-unicast-flood off
1426 bridge-multicast-flood off
1427
1428 auto vmbr2
1429 iface vmbr2 inet static
1430 bridge_ports vxlan2
1431 bridge_stp off
1432 bridge_fd 0
1433 address 10.0.2.254
1434 netmask 255.255.255.0
1435 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1436 vrf vrf1
1437
1438 auto vxlan3
1439 iface vxlan3 inet manual
1440 vxlan-id 3
1441 vxlan-local-tunnelip 192.168.0.3
1442 bridge-learning off
1443 bridge-arp-nd-suppress on
1444 bridge-unicast-flood off
1445 bridge-multicast-flood off
1446
1447 auto vmbr3
1448 iface vmbr3 inet static
1449 bridge_ports vxlan3
1450 bridge_stp off
1451 bridge_fd 0
1452 address 10.0.3.254
1453 netmask 255.255.255.0
1454 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1455 vrf vrf1
1456
1457 #interconnect vxlan-vfr l3vni
1458 auto vxlan4000
1459 iface vxlan4000 inet manual
1460 vxlan-id 4000
1461 vxlan-local-tunnelip 192.168.0.3
1462 bridge-learning off
1463 bridge-arp-nd-suppress on
1464 bridge-unicast-flood off
1465 bridge-multicast-flood off
1466
1467
1468 auto vmbr4000
1469 iface vmbr4000 inet manual
1470 bridge_ports vxlan4000
1471 bridge_stp off
1472 bridge_fd 0
1473 hwaddress 44:39:39:FF:40:92 #must be different on each node
1474 vrf vrf1
1475 ----
1476
1477
1478 frr.conf
1479
1480 ----
1481 vrf vrf1
1482 vni 4000
1483 !
1484 router bgp 1234
1485 bgp router-id 192.168.0.3
1486 no bgp default ipv4-unicast
1487 coalesce-time 1000
1488 neighbor 192.168.0.1 remote-as 1234
1489 neighbor 192.168.0.2 remote-as 1234
1490 !
1491 address-family l2vpn evpn
1492 neighbor 192.168.0.1 activate
1493 neighbor 192.168.0.2 activate
1494 advertise-all-vni
1495 exit-address-family
1496 !
1497 router bgp 1234 vrf vrf1
1498 !
1499 bgp router-id 192.168.0.3
1500 !
1501 address-family ipv4 unicast
1502 redistribute connected
1503 exit-address-family
1504 !
1505 address-family l2vpn evpn
1506 advertise ipv4 unicast
1507 exit-address-family
1508 !
1509 line vty
1510 !
1511 ----
1512
1513 multiple gateway nodes
1514 ^^^^^^^^^^^^^^^^^^^^^^
1515 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1516 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1517 and announce this default gw.
1518 The external router have ecmp routes to all proxmox nodes.(balancing).
1519 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1520 vxlan the packet to final destination.
1521
1522 *node1
1523
1524 ----
1525 auto vrf1
1526 iface vrf1
1527 vrf-table auto
1528
1529 auto eno1
1530 iface eno1 inet manual
1531 mtu 1550
1532
1533 auto vmbr0
1534 iface vmbr0 inet static
1535 address 192.168.0.1
1536 netmask 255.255.255.0
1537 bridge_ports eno1
1538 bridge_stp off
1539 bridge_fd 0
1540
1541 auto eno2
1542 iface eno2
1543 address 172.16.0.1
1544 netmask 255.255.255.0
1545 vrf vrf1
1546 mtu 1550
1547 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1548 #if you have multiple external routers, you can use ecmp balancing
1549 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1550
1551 auto vxlan2
1552 iface vxlan2 inet manual
1553 vxlan-id 2
1554 vxlan-local-tunnelip 192.168.0.1
1555 bridge-learning off
1556 bridge-arp-nd-suppress on
1557 bridge-unicast-flood off
1558 bridge-multicast-flood off
1559
1560 auto vmbr2
1561 iface vmbr2 inet static
1562 bridge_ports vxlan2
1563 bridge_stp off
1564 bridge_fd 0
1565 address 10.0.2.254
1566 netmask 255.255.255.0
1567 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1568 vrf vrf1
1569
1570 auto vxlan3
1571 iface vxlan3 inet manual
1572 vxlan-id 3
1573 vxlan-local-tunnelip 192.168.0.1
1574 bridge-learning off
1575 bridge-arp-nd-suppress on
1576 bridge-unicast-flood off
1577 bridge-multicast-flood off
1578
1579 auto vmbr3
1580 iface vmbr3 inet static
1581 bridge_ports vxlan3
1582 bridge_stp off
1583 bridge_fd 0
1584 address 10.0.3.254
1585 netmask 255.255.255.0
1586 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1587 vrf vrf1
1588
1589 #interconnect vxlan-vfr l3vni
1590 auto vxlan4000
1591 iface vxlan4000 inet manual
1592 vxlan-id 4000
1593 vxlan-local-tunnelip 192.168.0.1
1594 bridge-learning off
1595 bridge-arp-nd-suppress on
1596 bridge-unicast-flood off
1597 bridge-multicast-flood off
1598
1599 auto vmbr4000
1600 iface vmbr4000 inet manual
1601 bridge_ports vxlan4000
1602 bridge_stp off
1603 bridge_fd 0
1604 hwaddress 44:39:39:FF:40:90 #must be different on each node
1605 vrf vrf1
1606 ----
1607
1608
1609 frr.conf
1610
1611 ----
1612 vrf vrf1
1613 vni 4000
1614 !
1615 router bgp 1234
1616 bgp router-id 192.168.0.1
1617 no bgp default ipv4-unicast
1618 coalesce-time 1000
1619 neighbor 192.168.0.2 remote-as 1234
1620 neighbor 192.168.0.3 remote-as 1234
1621 !
1622 address-family l2vpn evpn
1623 neighbor 192.168.0.2 activate
1624 neighbor 192.168.0.3 activate
1625 advertise-all-vni
1626 exit-address-family
1627 !
1628 router bgp 1234 vrf vrf1
1629 !
1630 bgp router-id 172.16.0.1
1631 !
1632 address-family ipv4 unicast
1633 redistribute connected
1634 redistribute kernel !announce your default gw to all nodes
1635 exit-address-family
1636 !
1637 address-family l2vpn evpn
1638 advertise ipv4 unicast
1639 exit-address-family
1640 !
1641 line vty
1642 !
1643 ----
1644
1645
1646 * node2
1647
1648 ----
1649 auto vrf1
1650 iface vrf1
1651 vrf-table auto
1652
1653 auto eno1
1654 iface eno1 inet manual
1655 mtu 1550
1656
1657 auto vmbr0
1658 iface vmbr0 inet static
1659 address 192.168.0.2
1660 netmask 255.255.255.0
1661 bridge_ports eno1
1662 bridge_stp off
1663 bridge_fd 0
1664
1665 auto eno2
1666 iface eno2
1667 address 172.16.0.3
1668 netmask 255.255.255.0
1669 vrf vrf1
1670 mtu 1550
1671 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1672 #if you have multiple external routers, you can use ecmp balancing
1673 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1674
1675 auto vxlan2
1676 iface vxlan2 inet manual
1677 vxlan-id 2
1678 vxlan-local-tunnelip 192.168.0.2
1679 bridge-learning off
1680 bridge-arp-nd-suppress on
1681 bridge-unicast-flood off
1682 bridge-multicast-flood off
1683
1684 auto vmbr2
1685 iface vmbr2 inet static
1686 bridge_ports vxlan2
1687 bridge_stp off
1688 bridge_fd 0
1689 address 10.0.2.254
1690 netmask 255.255.255.0
1691 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1692 vrf vrf1
1693
1694 auto vxlan3
1695 iface vxlan3 inet manual
1696 vxlan-id 3
1697 vxlan-local-tunnelip 192.168.0.2
1698 bridge-learning off
1699 bridge-arp-nd-suppress on
1700 bridge-unicast-flood off
1701 bridge-multicast-flood off
1702
1703 auto vmbr3
1704 iface vmbr3 inet static
1705 bridge_ports vxlan3
1706 bridge_stp off
1707 bridge_fd 0
1708 address 10.0.3.254
1709 netmask 255.255.255.0
1710 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1711 vrf vrf1
1712
1713 #interconnect vxlan-vfr l3vni
1714 auto vxlan4000
1715 iface vxlan4000 inet manual
1716 vxlan-id 4000
1717 vxlan-local-tunnelip 192.168.0.2
1718 bridge-learning off
1719 bridge-arp-nd-suppress on
1720 bridge-unicast-flood off
1721 bridge-multicast-flood off
1722
1723
1724 auto vmbr4000
1725 iface vmbr4000 inet manual
1726 bridge_ports vxlan4000
1727 bridge_stp off
1728 bridge_fd 0
1729 hwaddress 44:39:39:FF:40:91 #must be different on each node
1730 vrf vrf1
1731 ----
1732
1733
1734 frr.conf
1735
1736 ----
1737 vrf vrf1
1738 vni 4000
1739 !
1740 router bgp 1234
1741 bgp router-id 192.168.0.2
1742 no bgp default ipv4-unicast
1743 coalesce-time 1000
1744 neighbor 192.168.0.1 remote-as 1234
1745 neighbor 192.168.0.3 remote-as 1234
1746 !
1747 address-family l2vpn evpn
1748 neighbor 192.168.0.1 activate
1749 neighbor 192.168.0.3 activate
1750 advertise-all-vni
1751 exit-address-family
1752 !
1753 router bgp 1234 vrf vrf1
1754 !
1755 bgp router-id 172.16.0.2
1756 !
1757 address-family ipv4 unicast
1758 redistribute connected
1759 redistribute kernel !announce your default gw to all nodes
1760 exit-address-family
1761 !
1762 address-family l2vpn evpn
1763 advertise ipv4 unicast
1764 exit-address-family
1765 !
1766 line vty
1767 !
1768 ----
1769
1770
1771 * node3
1772
1773 ----
1774 auto vrf1
1775 iface vrf1
1776 vrf-table auto
1777
1778 auto eno1
1779 iface eno1 inet manual
1780 mtu 1550
1781
1782 auto vmbr0
1783 iface vmbr0 inet static
1784 address 192.168.0.3
1785 netmask 255.255.255.0
1786 bridge_ports eno1
1787 bridge_stp off
1788 bridge_fd 0
1789
1790 auto eno2
1791 iface eno2
1792 address 172.16.0.3
1793 netmask 255.255.255.0
1794 vrf vrf1
1795 mtu 1550
1796 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1797 #if you have multiple external routers, you can use ecmp balancing
1798 #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1799
1800 auto vxlan2
1801 iface vxlan2 inet manual
1802 vxlan-id 2
1803 vxlan-local-tunnelip 192.168.0.3
1804 bridge-learning off
1805 bridge-arp-nd-suppress on
1806 bridge-unicast-flood off
1807 bridge-multicast-flood off
1808
1809 auto vmbr2
1810 iface vmbr2 inet static
1811 bridge_ports vxlan2
1812 bridge_stp off
1813 bridge_fd 0
1814 address 10.0.2.254
1815 netmask 255.255.255.0
1816 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1817 vrf vrf1
1818
1819 auto vxlan3
1820 iface vxlan3 inet manual
1821 vxlan-id 3
1822 vxlan-local-tunnelip 192.168.0.3
1823 bridge-learning off
1824 bridge-arp-nd-suppress on
1825 bridge-unicast-flood off
1826 bridge-multicast-flood off
1827
1828 auto vmbr3
1829 iface vmbr3 inet static
1830 bridge_ports vxlan3
1831 bridge_stp off
1832 bridge_fd 0
1833 address 10.0.3.254
1834 netmask 255.255.255.0
1835 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1836 vrf vrf1
1837
1838 #interconnect vxlan-vfr l3vni
1839 auto vxlan4000
1840 iface vxlan4000 inet manual
1841 vxlan-id 4000
1842 vxlan-local-tunnelip 192.168.0.3
1843 bridge-learning off
1844 bridge-arp-nd-suppress on
1845 bridge-unicast-flood off
1846 bridge-multicast-flood off
1847
1848
1849 auto vmbr4000
1850 iface vmbr4000 inet manual
1851 bridge_ports vxlan4000
1852 bridge_stp off
1853 bridge_fd 0
1854 hwaddress 44:39:39:FF:40:92 #must be different on each node
1855 vrf vrf1
1856 ----
1857
1858
1859 frr.conf
1860
1861 ----
1862 vrf vrf1
1863 vni 4000
1864 !
1865 router bgp 1234
1866 bgp router-id 192.168.0.3
1867 no bgp default ipv4-unicast
1868 coalesce-time 1000
1869 neighbor 192.168.0.1 remote-as 1234
1870 neighbor 192.168.0.2 remote-as 1234
1871 !
1872 address-family l2vpn evpn
1873 neighbor 192.168.0.1 activate
1874 neighbor 192.168.0.2 activate
1875 advertise-all-vni
1876 exit-address-family
1877 !
1878 router bgp 1234 vrf vrf1
1879 !
1880 bgp router-id 172.16.0.3
1881 !
1882 address-family ipv4 unicast
1883 redistribute connected
1884 redistribute kernel !announce your default gw to all nodes
1885 exit-address-family
1886 !
1887 address-family l2vpn evpn
1888 advertise ipv4 unicast
1889 exit-address-family
1890 !
1891 line vty
1892 !
1893 ----
1894
1895 Note
1896 ^^^^
1897
1898 If your external router don't support ecmp to reach multiple proxmox nodes,
1899 you can setup an HA floating vip on proxmox nodes with vrrp
1900
1901 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1902 Node1 is the primary and failover to node2 in case of failure.
1903
1904
1905 * node1
1906
1907 ----
1908 auto eno2
1909 iface eno2
1910 address 172.16.0.1
1911 netmask 255.255.255.0
1912 vrf vrf1
1913 mtu 1550
1914 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1915 vrrp-id 1
1916 vrrp-priority 1
1917 vrrp-virtual-ip 172.16.0.10
1918 ----
1919
1920 * node2
1921
1922 ----
1923 auto eno2
1924 iface eno2
1925 address 172.16.0.2
1926 netmask 255.255.255.0
1927 mtu 1550
1928 vrf vrf1
1929 post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1930 vrrp-id 1
1931 vrrp-priority 2
1932 vrrp-virtual-ip 172.16.0.10
1933 ----
1934
1935