split sysctl.conf to symmetric and asymmetric setup
[pve-docs.git] / vxlan-and-evpn.adoc
1
2 ////
3
4 This is currently not included, because
5 - it requires ifupdown2
6 - routing needs more documentation
7
8 ////
9
10
11 VXLAN layer2 with vlan unware linux bridges
12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
13
14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
15 while accommodating a very large number of tenants. It is defined in RFC 7348.
16 Each overlay network is known as a VXLAN Segment and identified by a unique
17 24-bit segment ID called a VXLAN Network Identifier (VNI).
18
19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
21
22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
23
24 multicast mode
25 ^^^^^^^^^^^^^^
26
27 This scenario relies in head end replication, meaning that end host in case
28 of not having any entry for the destination MAC address will send out an ARP
29 to other devices / VTEPs in the VXLAN network.
30 This is done by sending the request to the VXLAN multicast group,
31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
32
33
34 * node1
35
36 ----
37 auto eno1
38 iface eno1 inet manual
39
40 auto vmbr0
41 iface vmbr0 inet static
42 address 192.168.0.1
43 netmask 255.255.255.0
44 bridge_ports eno1
45 bridge_stp off
46 bridge_fd 0
47
48 auto vxlan2
49 iface vxlan2 inet manual
50 vxlan-svcnodeip 225.20.1.1
51 vxlan-physdev eno1
52
53 auto vmbr2
54 iface vmbr2 inet manual
55 bridge_ports vxlan2
56 bridge_stp off
57 bridge_fd 0
58
59 auto vxlan3
60 iface vxlan3 inet manual
61 vxlan-svcnodeip 225.20.1.1
62 vxlan-physdev eno1
63
64 auto vmbr3
65 iface vmbr3 inet manual
66 bridge_ports vxlan3
67 bridge_stp off
68 bridge_fd 0
69 ----
70
71
72 * node2
73
74 ----
75 auto eno1
76 iface eno1 inet manual
77
78 auto vmbr0
79 iface vmbr0 inet static
80 address 192.168.0.2
81 netmask 255.255.255.0
82 bridge_ports eno1
83 bridge_stp off
84 bridge_fd 0
85
86 auto vxlan2
87 iface vxlan2 inet manual
88 vxlan-svcnodeip 225.20.1.1
89 vxlan-physdev eno1
90
91 auto vmbr2
92 iface vmbr2 inet manual
93 bridge_ports vxlan2
94 bridge_stp off
95 bridge_fd 0
96
97
98 auto vxlan3
99 iface vxlan3 inet manual
100 vxlan-svcnodeip 225.20.1.1
101 vxlan-physdev eno1
102
103 auto vmbr3
104 iface vmbr3 inet manual
105 bridge_ports vxlan3
106 bridge_stp off
107 bridge_fd 0
108 ----
109
110
111 * node3
112
113 ----
114 auto eno1
115 iface eno1 inet manual
116
117 auto vmbr0
118 iface vmbr0 inet static
119 address 192.168.0.3
120 netmask 255.255.255.0
121 bridge_ports eno1
122 bridge_stp off
123 bridge_fd 0
124
125 auto vxlan2
126 iface vxlan2 inet manual
127 vxlan-svcnodeip 225.20.1.1
128 vxlan-physdev eno1
129
130 auto vmbr2
131 iface vmbr2 inet manual
132 bridge_ports vxlan2
133 bridge_stp off
134 bridge_fd 0
135
136
137 auto vxlan3
138 iface vxlan3 inet manual
139 vxlan-svcnodeip 225.20.1.1
140 vxlan-physdev eno1
141
142 auto vmbr3
143 iface vmbr3 inet manual
144 bridge_ports vxlan3
145 bridge_stp off
146 bridge_fd 0
147 ----
148
149
150 unicast mode
151 ^^^^^^^^^^^^
152
153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
154 The VXLAN is defined without a remote multicast group.
155 Instead, all the remote VTEPs are associated with the all-zero address:
156 a BUM frame will be duplicated to all these destinations.
157 The VXLAN device will still learn remote addresses automatically using source-address learning.
158
159 * node1
160
161 ----
162 auto eno1
163 iface eno1 inet manual
164
165 auto vmbr0
166 iface vmbr0 inet static
167 address 192.168.0.1
168 netmask 255.255.255.0
169 bridge_ports eno1
170 bridge_stp off
171 bridge_fd 0
172
173
174 auto vxlan2
175 iface vxlan2 inet manual
176 vxlan_remoteip 192.168.0.2
177 vxlan_remoteip 192.168.0.3
178
179
180 auto vmbr2
181 iface vmbr2 inet manual
182 bridge_ports vxlan2
183 bridge_stp off
184 bridge_fd 0
185
186
187 auto vxlan3
188 iface vxlan2 inet manual
189 vxlan_remoteip 192.168.0.2
190 vxlan_remoteip 192.168.0.3
191
192
193 auto vmbr3
194 iface vmbr3 inet manual
195 bridge_ports vxlan3
196 bridge_stp off
197 bridge_fd 0
198 ----
199
200
201 * node2
202
203 ----
204 auto eno1
205 iface eno1 inet manual
206
207 auto vmbr0
208 iface vmbr0 inet static
209 address 192.168.0.2
210 netmask 255.255.255.0
211 bridge_ports eno1
212 bridge_stp off
213 bridge_fd 0
214
215 auto vxlan2
216 iface vxlan2 inet manual
217 vxlan_remoteip 192.168.0.1
218 vxlan_remoteip 192.168.0.3
219
220
221
222 auto vmbr2
223 iface vmbr2 inet manual
224 bridge_ports vxlan2
225 bridge_stp off
226 bridge_fd 0
227
228 auto vxlan3
229 iface vxlan2 inet manual
230 vxlan_remoteip 192.168.0.1
231 vxlan_remoteip 192.168.0.3
232
233
234 auto vmbr3
235 iface vmbr3 inet manual
236 bridge_ports vxlan3
237 bridge_stp off
238 bridge_fd 0
239 ----
240
241
242 * node3
243
244 ----
245 auto eno1
246 iface eno1 inet manual
247
248 auto vmbr0
249 iface vmbr0 inet static
250 address 192.168.0.3
251 netmask 255.255.255.0
252 bridge_ports eno1
253 bridge_stp off
254 bridge_fd 0
255
256 auto vxlan2
257 iface vxlan2 inet manual
258 vxlan_remoteip 192.168.0.2
259 vxlan_remoteip 192.168.0.3
260
261
262
263 auto vmbr2
264 iface vmbr2 inet manual
265 bridge_ports vxlan2
266 bridge_stp off
267 bridge_fd 0
268
269 auto vxlan3
270 iface vxlan2 inet manual
271 vxlan_remoteip 192.168.0.2
272 vxlan_remoteip 192.168.0.3
273
274
275 auto vmbr3
276 iface vmbr3 inet manual
277 bridge_ports vxlan3
278 bridge_stp off
279 bridge_fd 0
280 ----
281
282
283 bgp-evpn
284 ^^^^^^^^
285
286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
288
289 The control plane used here is FRR, a bgp routing software.
290 Each node in the proxmox cluster peer with each others nodes.
291 For bigger networks, or multiple proxmox clusters,
292 it's possible to use external bgp route reflector servers.
293
294 * node1
295
296 ----
297 auto eno1
298 iface eno1 inet manual
299
300 auto vmbr0
301 iface vmbr0 inet static
302 address 192.168.0.1
303 netmask 255.255.255.0
304 bridge_ports eno1
305 bridge_stp off
306 bridge_fd 0
307
308 auto vxlan2
309 iface vxlan2 inet manual
310 vxlan-local-tunnelip 192.168.0.1
311 bridge-learning off
312 bridge-arp-nd-suppress on
313 bridge-unicast-flood off
314 bridge-multicast-flood off
315
316
317 auto vmbr2
318 iface vmbr2 inet manual
319 bridge_ports vxlan2
320 bridge_stp off
321 bridge_fd 0
322
323
324 auto vxlan3
325 iface vxlan3 inet manual
326 vxlan-local-tunnelip 192.168.0.1
327 bridge-learning off
328 bridge-arp-nd-suppress on
329 bridge-unicast-flood off
330 bridge-multicast-flood off
331
332
333 auto vmbr3
334 iface vmbr3 inet manual
335 bridge_ports vxlan3
336 bridge_stp off
337 bridge_fd 0
338 ----
339
340
341 /etc/frr/frr.conf
342
343 ----
344 router bgp 1234
345 no bgp default ipv4-unicast
346 coalesce-time 1000
347 neighbor 192.168.0.2 remote-as 1234
348 neighbor 192.168.0.3 remote-as 1234
349 !
350 address-family l2vpn evpn
351 neighbor 192.168.0.2 activate
352 neighbor 192.168.0.3 activate
353 advertise-all-vni
354 exit-address-family
355 !
356 line vty
357 !
358 ----
359
360
361 * node2
362
363 ----
364 auto eno1
365 iface eno1 inet manual
366
367 auto vmbr0
368 iface vmbr0 inet static
369 address 192.168.0.2
370 netmask 255.255.255.0
371 bridge_ports eno1
372 bridge_stp off
373 bridge_fd 0
374
375 auto vxlan2
376 iface vxlan2 inet manual
377 vxlan-local-tunnelip 192.168.0.2
378 bridge-learning off
379 bridge-arp-nd-suppress on
380 bridge-unicast-flood off
381 bridge-multicast-flood off
382
383
384 auto vmbr2
385 iface vmbr2 inet manual
386 bridge_ports vxlan2
387 bridge_stp off
388 bridge_fd 0
389
390 auto vxlan3
391 iface vxlan3 inet manual
392 vxlan-local-tunnelip 192.168.0.2
393 bridge-learning off
394 bridge-arp-nd-suppress on
395 bridge-unicast-flood off
396 bridge-multicast-flood off
397
398
399 auto vmbr3
400 iface vmbr3 inet manual
401 bridge_ports vxlan3
402 bridge_stp off
403 bridge_fd 0
404 ----
405
406
407 /etc/frr/frr.conf
408
409 ----
410 router bgp 1234
411 no bgp default ipv4-unicast
412 coalesce-time 1000
413 neighbor 192.168.0.1 remote-as 1234
414 neighbor 192.168.0.3 remote-as 1234
415 !
416 address-family l2vpn evpn
417 neighbor 192.168.0.1 activate
418 neighbor 192.168.0.3 activate
419 advertise-all-vni
420 exit-address-family
421 !
422 line vty
423 !
424 ----
425
426
427 * node3
428
429 ----
430 auto eno1
431 iface eno1 inet manual
432
433 auto vmbr0
434 iface vmbr0 inet static
435 address 192.168.0.2
436 netmask 255.255.255.0
437 bridge_ports eno1
438 bridge_stp off
439 bridge_fd 0
440
441 auto vxlan2
442 iface vxlan2 inet manual
443 vxlan-local-tunnelip 192.168.0.3
444 bridge-learning off
445 bridge-arp-nd-suppress on
446 bridge-unicast-flood off
447 bridge-multicast-flood off
448
449
450 auto vmbr2
451 iface vmbr2 inet manual
452 bridge_ports vxlan2
453 bridge_stp off
454 bridge_fd 0
455
456 auto vxlan3
457 iface vxlan3 inet manual
458 vxlan-local-tunnelip 192.168.0.3
459 bridge-learning off
460 bridge-arp-nd-suppress on
461 bridge-unicast-flood off
462 bridge-multicast-flood off
463
464
465 auto vmbr3
466 iface vmbr3 inet manual
467 bridge_ports vxlan3
468 bridge_stp off
469 bridge_fd 0
470 ----
471
472
473 /etc/frr/frr.conf
474
475
476 ----
477 router bgp 1234
478 no bgp default ipv4-unicast
479 coalesce-time 1000
480 neighbor 192.168.0.1 remote-as 1234
481 neighbor 192.168.0.2 remote-as 1234
482 !
483 address-family l2vpn evpn
484 neighbor 192.168.0.1 activate
485 neighbor 192.168.0.2 activate
486 advertise-all-vni
487 exit-address-family
488 !
489 line vty
490 !
491 ----
492
493 VXLAN layer3 routing with anycast gateway
494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
495
496 With this need, each vmbr bridge will be the gateway for the vm.
497 Same vmbr on different node, will have same ip address and same mac address,
498 to have working vm live migration and no network disruption.
499
500 VXLAN layer3 routing only work with FRR and non-aware bridge.
501 (vlan aware bridge support is buggy currently).
502
503 asymmetric model
504 ^^^^^^^^^^^^^^^^
505
506 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
507
508 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
509 but only bridging on the egress.
510 This results in bi-directional VXLAN traffic traveling on different VNIs
511 in each direction (always the destination VNI) across the routed infrastructure.
512
513 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
514
515
516 sysctl.conf tuning
517
518 ----
519 #enable routing
520 net.ipv4.ip_forward=1
521 net.ipv6.conf.all.forwarding=1
522 ----
523
524 * node1
525
526 ----
527 auto eno1
528 iface eno1 inet manual
529
530 auto vmbr0
531 iface vmbr0 inet static
532 address 192.168.0.1
533 netmask 255.255.255.0
534 bridge_ports eno1
535 bridge_stp off
536 bridge_fd 0
537
538 auto vxlan2
539 iface vxlan2 inet manual
540 vxlan-local-tunnelip 192.168.0.1
541 bridge-learning off
542 bridge-arp-nd-suppress on
543 bridge-unicast-flood off
544 bridge-multicast-flood off
545
546
547 auto vmbr2
548 iface vmbr2 inet static
549 address 10.0.2.254
550 netmask 255.255.255.0
551 hwaddress 44:39:39:FF:40:94
552 bridge_ports vxlan2
553 bridge_stp off
554 bridge_fd 0
555
556
557 auto vxlan3
558 iface vxlan3 inet manual
559 vxlan-local-tunnelip 192.168.0.1
560 bridge-learning off
561 bridge-arp-nd-suppress on
562 bridge-unicast-flood off
563 bridge-multicast-flood off
564
565
566 auto vmbr3
567 iface vmbr3 inet static
568 address 10.0.3.254
569 netmask 255.255.255.0
570 hwaddress 44:39:39:FF:40:94
571 bridge_ports vxlan3
572 bridge_stp off
573 bridge_fd 0
574 ----
575
576
577 frr.conf
578
579 ----
580 router bgp 1234
581 bgp router-id 192.168.0.1
582 no bgp default ipv4-unicast
583 coalesce-time 1000
584 neighbor 192.168.0.2 remote-as 1234
585 neighbor 192.168.0.3 remote-as 1234
586 !
587 address-family l2vpn evpn
588 neighbor 192.168.0.2 activate
589 neighbor 192.168.0.3 activate
590 advertise-all-vni
591 exit-address-family
592 !
593 line vty
594 !
595 ----
596
597
598 * node2
599
600 ----
601 auto eno1
602 iface eno1 inet manual
603
604 auto vmbr0
605 iface vmbr0 inet static
606 address 192.168.0.2
607 netmask 255.255.255.0
608 bridge_ports eno1
609 bridge_stp off
610 bridge_fd 0
611
612 auto vxlan2
613 iface vxlan2 inet manual
614 vxlan-local-tunnelip 192.168.0.2
615 bridge-learning off
616 bridge-arp-nd-suppress on
617 bridge-unicast-flood off
618 bridge-multicast-flood off
619
620
621 auto vmbr2
622 iface vmbr2 inet static
623 address 10.0.2.254
624 netmask 255.255.255.0
625 hwaddress 44:39:39:FF:40:94
626 bridge_ports vxlan2
627 bridge_stp off
628 bridge_fd 0
629
630
631 auto vxlan3
632 iface vxlan3 inet manual
633 vxlan-local-tunnelip 192.168.0.2
634 bridge-learning off
635 bridge-arp-nd-suppress on
636 bridge-unicast-flood off
637 bridge-multicast-flood off
638
639
640 auto vmbr3
641 iface vmbr3 inet static
642 address 10.0.3.254
643 netmask 255.255.255.0
644 hwaddress 44:39:39:FF:40:94
645 bridge_ports vxlan3
646 bridge_stp off
647 bridge_fd 0
648 ----
649
650
651 frr.conf
652
653 ----
654 router bgp 1234
655 bgp router-id 192.168.0.2
656 no bgp default ipv4-unicast
657 coalesce-time 1000
658 neighbor 192.168.0.1 remote-as 1234
659 neighbor 192.168.0.3 remote-as 1234
660 !
661 address-family l2vpn evpn
662 neighbor 192.168.0.1 activate
663 neighbor 192.168.0.3 activate
664 advertise-all-vni
665 exit-address-family
666 !
667 line vty
668 !
669 ----
670
671
672 * node3
673
674 ----
675 auto eno1
676 iface eno1 inet manual
677
678 auto vmbr0
679 iface vmbr0 inet static
680 address 192.168.0.3
681 netmask 255.255.255.0
682 bridge_ports eno1
683 bridge_stp off
684 bridge_fd 0
685
686 auto vxlan2
687 iface vxlan2 inet manual
688 vxlan-local-tunnelip 192.168.0.3
689 bridge-learning off
690 bridge-arp-nd-suppress on
691 bridge-unicast-flood off
692 bridge-multicast-flood off
693
694
695 auto vmbr2
696 iface vmbr2 inet static
697 address 10.0.2.254
698 netmask 255.255.255.0
699 hwaddress 44:39:39:FF:40:94
700 bridge_ports vxlan2
701 bridge_stp off
702 bridge_fd 0
703
704
705 auto vxlan3
706 iface vxlan3 inet manual
707 vxlan-local-tunnelip 192.168.0.3
708 bridge-learning off
709 bridge-arp-nd-suppress on
710 bridge-unicast-flood off
711 bridge-multicast-flood off
712
713
714 auto vmbr3
715 iface vmbr3 inet static
716 address 10.0.3.254
717 netmask 255.255.255.0
718 hwaddress 44:39:39:FF:40:94
719 bridge_ports vxlan3
720 bridge_stp off
721 bridge_fd 0
722 ----
723
724
725 frr.conf
726
727 ----
728 router bgp 1234
729 bgp router-id 192.168.0.3
730 no bgp default ipv4-unicast
731 coalesce-time 1000
732 neighbor 192.168.0.1 remote-as 1234
733 neighbor 192.168.0.2 remote-as 1234
734 !
735 address-family l2vpn evpn
736 neighbor 192.168.0.1 activate
737 neighbor 192.168.0.2 activate
738 advertise-all-vni
739 exit-address-family
740 !
741 line vty
742 !
743 ----
744
745
746 symmetric model
747 ^^^^^^^^^^^^^^^
748
749 With this model, you don't need to have all vxlan on all nodes.
750 This model will also be needed to route traffic to an external router.
751
752 The symmetric model routes and bridges on both the ingress and the egress leafs.
753 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
754 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
755 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
756 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
757
758 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
759
760 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
761
762 sysctl.conf tuning
763
764 ----
765 #enable routing
766 net.ipv4.ip_forward=1
767 net.ipv6.conf.all.forwarding=1
768 #disable reverse path filtering
769 net.ipv4.conf.default.rp_filter=0
770 net.ipv4.conf.all.rp_filter=0
771 #allow frr to work with vrf (kernel >4.14 bug)
772 net.ipv4.tcp_l3mdev_accept=1
773 ----
774
775 * node1
776
777 ----
778 auto vrf1
779 iface vrf1
780 vrf-table auto
781
782 auto eno1
783 iface eno1 inet manual
784
785 auto vmbr0
786 iface vmbr0 inet static
787 address 192.168.0.1
788 netmask 255.255.255.0
789 bridge_ports eno1
790 bridge_stp off
791 bridge_fd 0
792
793 auto vxlan2
794 iface vxlan2 inet manual
795 vxlan-local-tunnelip 192.168.0.1
796 bridge-learning off
797 bridge-arp-nd-suppress on
798 bridge-unicast-flood off
799 bridge-multicast-flood off
800
801 auto vmbr2
802 iface vmbr2 inet static
803 bridge_ports vxlan2
804 bridge_stp off
805 bridge_fd 0
806 address 10.0.2.254
807 netmask 255.255.255.0
808 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
809 vrf vrf1
810
811 auto vxlan3
812 iface vxlan3 inet manual
813 vxlan-local-tunnelip 192.168.0.1
814 bridge-learning off
815 bridge-arp-nd-suppress on
816 bridge-unicast-flood off
817 bridge-multicast-flood off
818
819 auto vmbr3
820 iface vmbr3 inet static
821 bridge_ports vxlan3
822 bridge_stp off
823 bridge_fd 0
824 address 10.0.3.254
825 netmask 255.255.255.0
826 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
827 vrf vrf1
828
829 #interconnect vxlan-vfr l3vni
830 auto vxlan4000
831 iface vxlan4000 inet manual
832 vxlan-local-tunnelip 192.168.0.1
833 bridge-learning off
834 bridge-arp-nd-suppress on
835 bridge-unicast-flood off
836 bridge-multicast-flood off
837
838
839 auto vmbr4000
840 iface vmbr4000 inet manual
841 bridge_ports vxlan4000
842 bridge_stp off
843 bridge_fd 0
844 hwaddress 44:39:39:FF:40:90 #must be different on each node
845 vrf vrf1
846 ----
847
848 frr.conf
849
850 ----
851 vrf vrf1
852 vni 4000
853 !
854 router bgp 1234
855 bgp router-id 192.168.0.1
856 no bgp default ipv4-unicast
857 coalesce-time 1000
858 neighbor 192.168.0.2 remote-as 1234
859 neighbor 192.168.0.3 remote-as 1234
860 !
861 address-family l2vpn evpn
862 neighbor 192.168.0.2 activate
863 neighbor 192.168.0.3 activate
864 advertise-all-vni
865 exit-address-family
866 !
867 router bgp 1234 vrf vrf1
868 !
869 bgp router-id 192.168.0.1
870 !
871 address-family ipv4 unicast
872 redistribute connected
873 exit-address-family
874 !
875 address-family l2vpn evpn
876 advertise ipv4 unicast
877 exit-address-family
878 !
879 line vty
880 !
881 ----
882
883
884 * node2
885
886 ----
887 auto vrf1
888 iface vrf1
889 vrf-table auto
890
891 auto eno1
892 iface eno1 inet manual
893
894 auto vmbr0
895 iface vmbr0 inet static
896 address 192.168.0.2
897 netmask 255.255.255.0
898 bridge_ports eno1
899 bridge_stp off
900 bridge_fd 0
901
902 auto vxlan2
903 iface vxlan2 inet manual
904 vxlan-local-tunnelip 192.168.0.2
905 bridge-learning off
906 bridge-arp-nd-suppress on
907 bridge-unicast-flood off
908 bridge-multicast-flood off
909
910 auto vmbr2
911 iface vmbr2 inet static
912 bridge_ports vxlan2
913 bridge_stp off
914 bridge_fd 0
915 address 10.0.2.254
916 netmask 255.255.255.0
917 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
918 vrf vrf1
919
920 auto vxlan3
921 iface vxlan3 inet manual
922 vxlan-local-tunnelip 192.168.0.2
923 bridge-learning off
924 bridge-arp-nd-suppress on
925 bridge-unicast-flood off
926 bridge-multicast-flood off
927
928 auto vmbr3
929 iface vmbr3 inet static
930 bridge_ports vxlan3
931 bridge_stp off
932 bridge_fd 0
933 address 10.0.3.254
934 netmask 255.255.255.0
935 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
936 vrf vrf1
937
938 #interconnect vxlan-vfr l3vni
939 auto vxlan4000
940 iface vxlan4000 inet manual
941 vxlan-local-tunnelip 192.168.0.2
942 bridge-learning off
943 bridge-arp-nd-suppress on
944 bridge-unicast-flood off
945 bridge-multicast-flood off
946
947
948 auto vmbr4000
949 iface vmbr4000 inet manual
950 bridge_ports vxlan4000
951 bridge_stp off
952 bridge_fd 0
953 hwaddress 44:39:39:FF:40:91 #must be different on each node
954 vrf vrf1
955 ----
956
957
958 frr.conf
959
960 ----
961 vrf vrf1
962 vni 4000
963 !
964 router bgp 1234
965 bgp router-id 192.168.0.2
966 no bgp default ipv4-unicast
967 coalesce-time 1000
968 neighbor 192.168.0.1 remote-as 1234
969 neighbor 192.168.0.3 remote-as 1234
970 !
971 address-family l2vpn evpn
972 neighbor 192.168.0.1 activate
973 neighbor 192.168.0.3 activate
974 advertise-all-vni
975 exit-address-family
976 !
977 router bgp 1234 vrf vrf1
978 !
979 bgp router-id 192.168.0.2
980 !
981 address-family ipv4 unicast
982 redistribute connected
983 exit-address-family
984 !
985 address-family l2vpn evpn
986 advertise ipv4 unicast
987 exit-address-family
988 !
989 line vty
990 !
991 ----
992
993
994 * node3
995
996 ----
997 auto vrf1
998 iface vrf1
999 vrf-table auto
1000
1001 auto eno1
1002 iface eno1 inet manual
1003
1004 auto vmbr0
1005 iface vmbr0 inet static
1006 address 192.168.0.3
1007 netmask 255.255.255.0
1008 bridge_ports eno1
1009 bridge_stp off
1010 bridge_fd 0
1011
1012 auto vxlan2
1013 iface vxlan2 inet manual
1014 vxlan-local-tunnelip 192.168.0.3
1015 bridge-learning off
1016 bridge-arp-nd-suppress on
1017 bridge-unicast-flood off
1018 bridge-multicast-flood off
1019
1020 auto vmbr2
1021 iface vmbr2 inet static
1022 bridge_ports vxlan2
1023 bridge_stp off
1024 bridge_fd 0
1025 address 10.0.2.254
1026 netmask 255.255.255.0
1027 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1028 vrf vrf1
1029
1030 auto vxlan3
1031 iface vxlan3 inet manual
1032 vxlan-local-tunnelip 192.168.0.3
1033 bridge-learning off
1034 bridge-arp-nd-suppress on
1035 bridge-unicast-flood off
1036 bridge-multicast-flood off
1037
1038 auto vmbr3
1039 iface vmbr3 inet static
1040 bridge_ports vxlan3
1041 bridge_stp off
1042 bridge_fd 0
1043 address 10.0.3.254
1044 netmask 255.255.255.0
1045 hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1046 vrf vrf1
1047
1048 #interconnect vxlan-vfr l3vni
1049 auto vxlan4000
1050 iface vxlan4000 inet manual
1051 vxlan-local-tunnelip 192.168.0.3
1052 bridge-learning off
1053 bridge-arp-nd-suppress on
1054 bridge-unicast-flood off
1055 bridge-multicast-flood off
1056
1057
1058 auto vmbr4000
1059 iface vmbr4000 inet manual
1060 bridge_ports vxlan4000
1061 bridge_stp off
1062 bridge_fd 0
1063 hwaddress 44:39:39:FF:40:92 #must be different on each node
1064 vrf vrf1
1065 ----
1066
1067
1068 frr.conf
1069
1070 ----
1071 vrf vrf1
1072 vni 4000
1073 !
1074 router bgp 1234
1075 bgp router-id 192.168.0.3
1076 no bgp default ipv4-unicast
1077 coalesce-time 1000
1078 neighbor 192.168.0.1 remote-as 1234
1079 neighbor 192.168.0.2 remote-as 1234
1080 !
1081 address-family l2vpn evpn
1082 neighbor 192.168.0.1 activate
1083 neighbor 192.168.0.2 activate
1084 advertise-all-vni
1085 exit-address-family
1086 !
1087 router bgp 1234 vrf vrf1
1088 !
1089 bgp router-id 192.168.0.3
1090 !
1091 address-family ipv4 unicast
1092 redistribute connected
1093 exit-address-family
1094 !
1095 address-family l2vpn evpn
1096 advertise ipv4 unicast
1097 exit-address-family
1098 !
1099 line vty
1100 !
1101 ----