vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  coalesce-time 1000
 371  neighbor 192.168.0.2 remote-as 1234
 372  neighbor 192.168.0.3 remote-as 1234
 373  !
 374  address-family l2vpn evpn
 375   neighbor 192.168.0.2 activate
 376   neighbor 192.168.0.3 activate
 377   advertise-all-vni
 378  exit-address-family
 379 !
 380 line vty
 381 !
 382 ----
 383
 384
 385 * node2
 386
 387 ----
 388 auto eno1
 389 iface eno1 inet manual
 390         mtu 1550
 391
 392 auto vmbr0
 393 iface vmbr0 inet static
 394         address 192.168.0.2
 395         netmask 255.255.255.0
 396         bridge_ports eno1
 397         bridge_stp off
 398         bridge_fd 0
 399
 400 auto vxlan2
 401 iface vxlan2 inet manual
 402         vxlan-id 2
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr2
 411 iface vmbr2 inet manual
 412         bridge_ports vxlan2
 413         bridge_stp off
 414         bridge_fd 0
 415
 416 auto vxlan3
 417 iface vxlan3 inet manual
 418         vxlan-id 3
 419         vxlan-local-tunnelip 192.168.0.2
 420         bridge-learning off
 421         bridge-arp-nd-suppress on
 422         bridge-unicast-flood off
 423         bridge-multicast-flood off
 424
 425
 426 auto vmbr3
 427 iface vmbr3 inet manual
 428         bridge_ports vxlan3
 429         bridge_stp off
 430         bridge_fd 0
 431 ----
 432
 433
 434 /etc/frr/frr.conf
 435
 436 ----
 437 router bgp 1234
 438  no bgp default ipv4-unicast
 439  coalesce-time 1000
 440  neighbor 192.168.0.1 remote-as 1234
 441  neighbor 192.168.0.3 remote-as 1234
 442  !
 443  address-family l2vpn evpn
 444   neighbor 192.168.0.1 activate
 445   neighbor 192.168.0.3 activate
 446   advertise-all-vni
 447  exit-address-family
 448 !
 449 line vty
 450 !
 451 ----
 452
 453
 454 * node3
 455
 456 ----
 457 auto eno1
 458 iface eno1 inet manual
 459         mtu 1550
 460
 461 auto vmbr0
 462 iface vmbr0 inet static
 463         address 192.168.0.2
 464         netmask 255.255.255.0
 465         bridge_ports eno1
 466         bridge_stp off
 467         bridge_fd 0
 468
 469 auto vxlan2
 470 iface vxlan2 inet manual
 471         vxlan-id 2
 472         vxlan-local-tunnelip 192.168.0.3
 473         bridge-learning off
 474         bridge-arp-nd-suppress on
 475         bridge-unicast-flood off
 476         bridge-multicast-flood off
 477
 478
 479 auto vmbr2
 480 iface vmbr2 inet manual
 481         bridge_ports vxlan2
 482         bridge_stp off
 483         bridge_fd 0
 484
 485 auto vxlan3
 486 iface vxlan3 inet manual
 487         vxlan-id 3
 488         vxlan-local-tunnelip 192.168.0.3
 489         bridge-learning off
 490         bridge-arp-nd-suppress on
 491         bridge-unicast-flood off
 492         bridge-multicast-flood off
 493
 494
 495 auto vmbr3
 496 iface vmbr3 inet manual
 497         bridge_ports vxlan3
 498         bridge_stp off
 499         bridge_fd 0
 500 ----
 501
 502
 503 /etc/frr/frr.conf
 504
 505
 506 ----
 507 router bgp 1234
 508  no bgp default ipv4-unicast
 509  coalesce-time 1000
 510  neighbor 192.168.0.1 remote-as 1234
 511  neighbor 192.168.0.2 remote-as 1234
 512  !
 513  address-family l2vpn evpn
 514   neighbor 192.168.0.1 activate
 515   neighbor 192.168.0.2 activate
 516   advertise-all-vni
 517  exit-address-family
 518 !
 519 line vty
 520 !
 521 ----
 522
 523 VXLAN layer3 routing with anycast gateway
 524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 525
 526 With this need, each vmbr bridge will be the gateway for the vm.
 527 Same vmbr on different node, will have same ip address and same mac address,
 528 to have working vm live migration and no network disruption.
 529
 530 VXLAN layer3 routing only work with FRR and non-aware bridge.
 531 (vlan aware bridge support is buggy currently).
 532
 533 asymmetric model
 534 ^^^^^^^^^^^^^^^^
 535
 536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 537
 538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 539 but only bridging on the egress.
 540 This results in bi-directional VXLAN traffic traveling on different VNIs
 541 in each direction (always the destination VNI) across the routed infrastructure.
 542
 543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 544
 545
 546 sysctl.conf tuning
 547
 548 ----
 549 #enable routing
 550 net.ipv4.ip_forward=1
 551 net.ipv6.conf.all.forwarding=1
 552 ----
 553
 554 * node1
 555
 556 ----
 557 auto eno1
 558 iface eno1 inet manual
 559         mtu 1550
 560
 561 auto vmbr0
 562 iface vmbr0 inet static
 563          address 192.168.0.1
 564          netmask  255.255.255.0
 565          bridge_ports eno1
 566          bridge_stp off
 567          bridge_fd 0
 568
 569 auto vxlan2
 570 iface vxlan2 inet manual
 571         vxlan-id 2
 572         vxlan-local-tunnelip 192.168.0.1
 573         bridge-learning off
 574         bridge-arp-nd-suppress on
 575         bridge-unicast-flood off
 576         bridge-multicast-flood off
 577
 578
 579 auto vmbr2
 580 iface vmbr2 inet static
 581         address 10.0.2.254
 582         netmask 255.255.255.0
 583         hwaddress 44:39:39:FF:40:94
 584         bridge_ports vxlan2
 585         bridge_stp off
 586         bridge_fd 0
 587
 588
 589 auto vxlan3
 590 iface vxlan3 inet manual
 591         vxlan-id 3
 592         vxlan-local-tunnelip 192.168.0.1
 593         bridge-learning off
 594         bridge-arp-nd-suppress on
 595         bridge-unicast-flood off
 596         bridge-multicast-flood off
 597
 598
 599 auto vmbr3
 600 iface vmbr3 inet static
 601         address 10.0.3.254
 602         netmask 255.255.255.0
 603         hwaddress 44:39:39:FF:40:94
 604         bridge_ports vxlan3
 605         bridge_stp off
 606         bridge_fd 0
 607 ----
 608
 609
 610 frr.conf
 611
 612 ----
 613 router bgp 1234
 614  bgp router-id 192.168.0.1
 615  no bgp default ipv4-unicast
 616  coalesce-time 1000
 617  neighbor 192.168.0.2 remote-as 1234
 618  neighbor 192.168.0.3 remote-as 1234
 619  !
 620  address-family l2vpn evpn
 621   neighbor 192.168.0.2 activate
 622   neighbor 192.168.0.3 activate
 623   advertise-all-vni
 624  exit-address-family
 625 !
 626 line vty
 627 !
 628 ----
 629
 630
 631 * node2
 632
 633 ----
 634 auto eno1
 635 iface eno1 inet manual
 636         mtu 1550
 637
 638 auto vmbr0
 639 iface vmbr0 inet static
 640          address 192.168.0.2
 641          netmask  255.255.255.0
 642          bridge_ports eno1
 643          bridge_stp off
 644          bridge_fd 0
 645
 646 auto vxlan2
 647 iface vxlan2 inet manual
 648         vxlan-id 2
 649         vxlan-local-tunnelip 192.168.0.2
 650         bridge-learning off
 651         bridge-arp-nd-suppress on
 652         bridge-unicast-flood off
 653         bridge-multicast-flood off
 654
 655
 656 auto vmbr2
 657 iface vmbr2 inet static
 658         address 10.0.2.254
 659         netmask 255.255.255.0
 660         hwaddress 44:39:39:FF:40:94
 661         bridge_ports vxlan2
 662         bridge_stp off
 663         bridge_fd 0
 664
 665
 666 auto vxlan3
 667 iface vxlan3 inet manual
 668         vxlan-id 3
 669         vxlan-local-tunnelip 192.168.0.2
 670         bridge-learning off
 671         bridge-arp-nd-suppress on
 672         bridge-unicast-flood off
 673         bridge-multicast-flood off
 674
 675
 676 auto vmbr3
 677 iface vmbr3 inet static
 678         address 10.0.3.254
 679         netmask 255.255.255.0
 680         hwaddress 44:39:39:FF:40:94
 681         bridge_ports vxlan3
 682         bridge_stp off
 683         bridge_fd 0
 684 ----
 685
 686
 687 frr.conf
 688
 689 ----
 690 router bgp 1234
 691  bgp router-id 192.168.0.2
 692  no bgp default ipv4-unicast
 693  coalesce-time 1000
 694  neighbor 192.168.0.1 remote-as 1234
 695  neighbor 192.168.0.3 remote-as 1234
 696  !
 697  address-family l2vpn evpn
 698   neighbor 192.168.0.1 activate
 699   neighbor 192.168.0.3 activate
 700   advertise-all-vni
 701  exit-address-family
 702 !
 703 line vty
 704 !
 705 ----
 706
 707
 708 * node3
 709
 710 ----
 711 auto eno1
 712 iface eno1 inet manual
 713         mtu 1550
 714
 715 auto vmbr0
 716 iface vmbr0 inet static
 717          address 192.168.0.3
 718          netmask  255.255.255.0
 719          bridge_ports eno1
 720          bridge_stp off
 721          bridge_fd 0
 722
 723 auto vxlan2
 724 iface vxlan2 inet manual
 725         vxlan-id 2
 726         vxlan-local-tunnelip 192.168.0.3
 727         bridge-learning off
 728         bridge-arp-nd-suppress on
 729         bridge-unicast-flood off
 730         bridge-multicast-flood off
 731
 732
 733 auto vmbr2
 734 iface vmbr2 inet static
 735         address 10.0.2.254
 736         netmask 255.255.255.0
 737         hwaddress 44:39:39:FF:40:94
 738         bridge_ports vxlan2
 739         bridge_stp off
 740         bridge_fd 0
 741
 742
 743 auto vxlan3
 744 iface vxlan3 inet manual
 745         vxlan-id 3
 746         vxlan-local-tunnelip 192.168.0.3
 747         bridge-learning off
 748         bridge-arp-nd-suppress on
 749         bridge-unicast-flood off
 750         bridge-multicast-flood off
 751
 752
 753 auto vmbr3
 754 iface vmbr3 inet static
 755         address 10.0.3.254
 756         netmask 255.255.255.0
 757         hwaddress 44:39:39:FF:40:94
 758         bridge_ports vxlan3
 759         bridge_stp off
 760         bridge_fd 0
 761 ----
 762
 763
 764 frr.conf
 765
 766 ----
 767 router bgp 1234
 768  bgp router-id 192.168.0.3
 769  no bgp default ipv4-unicast
 770  coalesce-time 1000
 771  neighbor 192.168.0.1 remote-as 1234
 772  neighbor 192.168.0.2 remote-as 1234
 773  !
 774  address-family l2vpn evpn
 775   neighbor 192.168.0.1 activate
 776   neighbor 192.168.0.2 activate
 777   advertise-all-vni
 778  exit-address-family
 779 !
 780 line vty
 781 !
 782 ----
 783
 784
 785 symmetric model
 786 ^^^^^^^^^^^^^^^
 787
 788 With this model, you don't need to have all vxlan on all nodes.
 789 This model will also be needed to route traffic to an external router.
 790
 791 The symmetric model routes and bridges on both the ingress and the egress leafs.
 792 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 793 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 794 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 795 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 796
 797 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 798
 799 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 800
 801 sysctl.conf tuning
 802
 803 ----
 804 #enable routing
 805 net.ipv4.ip_forward=1
 806 net.ipv6.conf.all.forwarding=1
 807 ----
 808
 809 * node1
 810
 811 ----
 812 auto vrf1
 813 iface vrf1
 814     vrf-table auto
 815
 816 auto eno1
 817 iface eno1 inet manual
 818         mtu 1550
 819
 820 auto vmbr0
 821 iface vmbr0 inet static
 822          address 192.168.0.1
 823          netmask  255.255.255.0
 824          bridge_ports eno1
 825          bridge_stp off
 826          bridge_fd 0
 827
 828 auto vxlan2
 829 iface vxlan2 inet manual
 830         vxlan-id 2
 831         vxlan-local-tunnelip 192.168.0.1
 832         bridge-learning off
 833         bridge-arp-nd-suppress on
 834         bridge-unicast-flood off
 835         bridge-multicast-flood off
 836
 837 auto vmbr2
 838 iface vmbr2 inet static
 839         bridge_ports vxlan2
 840         bridge_stp off
 841         bridge_fd 0
 842         address 10.0.2.254
 843         netmask 255.255.255.0
 844         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 845         vrf vrf1
 846
 847 auto vxlan3
 848 iface vxlan3 inet manual
 849         vxlan-id 3
 850         vxlan-local-tunnelip 192.168.0.1
 851         bridge-learning off
 852         bridge-arp-nd-suppress on
 853         bridge-unicast-flood off
 854         bridge-multicast-flood off
 855
 856 auto vmbr3
 857 iface vmbr3 inet static
 858         bridge_ports vxlan3
 859         bridge_stp off
 860         bridge_fd 0
 861         address 10.0.3.254
 862         netmask 255.255.255.0
 863         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 864         vrf vrf1
 865
 866 #interconnect vxlan-vfr l3vni
 867 auto vxlan4000
 868 iface vxlan4000 inet manual
 869         vxlan-id 4000
 870         vxlan-local-tunnelip 192.168.0.1
 871         bridge-learning off
 872         bridge-arp-nd-suppress on
 873         bridge-unicast-flood off
 874         bridge-multicast-flood off
 875
 876
 877 auto vmbr4000
 878 iface vmbr4000 inet manual
 879         bridge_ports vxlan4000
 880         bridge_stp off
 881         bridge_fd 0
 882         hwaddress 44:39:39:FF:40:90  #must be different on each node
 883         vrf vrf1
 884 ----
 885
 886 frr.conf
 887
 888 ----
 889 vrf vrf1
 890  vni 4000
 891 !
 892 router bgp 1234
 893  bgp router-id 192.168.0.1
 894  no bgp default ipv4-unicast
 895  coalesce-time 1000
 896  neighbor 192.168.0.2 remote-as 1234
 897  neighbor 192.168.0.3 remote-as 1234
 898  !
 899  address-family l2vpn evpn
 900   neighbor 192.168.0.2 activate
 901   neighbor 192.168.0.3 activate
 902   advertise-all-vni
 903  exit-address-family
 904 !
 905 router bgp 1234 vrf vrf1
 906 !
 907  bgp router-id 192.168.0.1
 908  !
 909  address-family ipv4 unicast
 910   redistribute connected
 911  exit-address-family
 912  !
 913  address-family l2vpn evpn
 914   advertise ipv4 unicast
 915  exit-address-family
 916 !
 917 line vty
 918 !
 919 ----
 920
 921
 922 * node2
 923
 924 ----
 925 auto vrf1
 926 iface vrf1
 927     vrf-table auto
 928
 929 auto eno1
 930 iface eno1 inet manual
 931         mtu 1550
 932
 933 auto vmbr0
 934 iface vmbr0 inet static
 935          address 192.168.0.2
 936          netmask  255.255.255.0
 937          bridge_ports eno1
 938          bridge_stp off
 939          bridge_fd 0
 940
 941 auto vxlan2
 942 iface vxlan2 inet manual
 943         vxlan-id 2
 944         vxlan-local-tunnelip 192.168.0.2
 945         bridge-learning off
 946         bridge-arp-nd-suppress on
 947         bridge-unicast-flood off
 948         bridge-multicast-flood off
 949
 950 auto vmbr2
 951 iface vmbr2 inet static
 952         bridge_ports vxlan2
 953         bridge_stp off
 954         bridge_fd 0
 955         address 10.0.2.254
 956         netmask 255.255.255.0
 957         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 958         vrf vrf1
 959
 960 auto vxlan3
 961 iface vxlan3 inet manual
 962         vxlan-id 3
 963         vxlan-local-tunnelip 192.168.0.2
 964         bridge-learning off
 965         bridge-arp-nd-suppress on
 966         bridge-unicast-flood off
 967         bridge-multicast-flood off
 968
 969 auto vmbr3
 970 iface vmbr3 inet static
 971         bridge_ports vxlan3
 972         bridge_stp off
 973         bridge_fd 0
 974         address 10.0.3.254
 975         netmask 255.255.255.0
 976         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 977         vrf vrf1
 978
 979 #interconnect vxlan-vfr l3vni
 980 auto vxlan4000
 981 iface vxlan4000 inet manual
 982         vxlan-id 4000
 983         vxlan-local-tunnelip 192.168.0.2
 984         bridge-learning off
 985         bridge-arp-nd-suppress on
 986         bridge-unicast-flood off
 987         bridge-multicast-flood off
 988
 989
 990 auto vmbr4000
 991 iface vmbr4000 inet manual
 992         bridge_ports vxlan4000
 993         bridge_stp off
 994         bridge_fd 0
 995         hwaddress 44:39:39:FF:40:91  #must be different on each node
 996         vrf vrf1
 997 ----
 998
 999
1000 frr.conf
1001
1002 ----
1003 vrf vrf1
1004  vni 4000
1005 !
1006 router bgp 1234
1007  bgp router-id 192.168.0.2
1008  no bgp default ipv4-unicast
1009  coalesce-time 1000
1010  neighbor 192.168.0.1 remote-as 1234
1011  neighbor 192.168.0.3 remote-as 1234
1012  !
1013  address-family l2vpn evpn
1014   neighbor 192.168.0.1 activate
1015   neighbor 192.168.0.3 activate
1016   advertise-all-vni
1017  exit-address-family
1018 !
1019 router bgp 1234 vrf vrf1
1020 !
1021  bgp router-id 192.168.0.2
1022  !
1023  address-family ipv4 unicast
1024   redistribute connected
1025  exit-address-family
1026  !
1027  address-family l2vpn evpn
1028   advertise ipv4 unicast
1029  exit-address-family
1030 !
1031 line vty
1032 !
1033 ----
1034
1035
1036 * node3
1037
1038 ----
1039 auto vrf1
1040 iface vrf1
1041     vrf-table auto
1042
1043 auto eno1
1044 iface eno1 inet manual
1045         mtu 1550
1046
1047 auto vmbr0
1048 iface vmbr0 inet static
1049          address 192.168.0.3
1050          netmask  255.255.255.0
1051          bridge_ports eno1
1052          bridge_stp off
1053          bridge_fd 0
1054
1055 auto vxlan2
1056 iface vxlan2 inet manual
1057         vxlan-id 2
1058         vxlan-local-tunnelip 192.168.0.3
1059         bridge-learning off
1060         bridge-arp-nd-suppress on
1061         bridge-unicast-flood off
1062         bridge-multicast-flood off
1063
1064 auto vmbr2
1065 iface vmbr2 inet static
1066         bridge_ports vxlan2
1067         bridge_stp off
1068         bridge_fd 0
1069         address 10.0.2.254
1070         netmask 255.255.255.0
1071         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1072         vrf vrf1
1073
1074 auto vxlan3
1075 iface vxlan3 inet manual
1076         vxlan-id 3
1077         vxlan-local-tunnelip 192.168.0.3
1078         bridge-learning off
1079         bridge-arp-nd-suppress on
1080         bridge-unicast-flood off
1081         bridge-multicast-flood off
1082
1083 auto vmbr3
1084 iface vmbr3 inet static
1085         bridge_ports vxlan3
1086         bridge_stp off
1087         bridge_fd 0
1088         address 10.0.3.254
1089         netmask 255.255.255.0
1090         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1091         vrf vrf1
1092
1093 #interconnect vxlan-vfr l3vni
1094 auto vxlan4000
1095 iface vxlan4000 inet manual
1096         vxlan-id 4000
1097         vxlan-local-tunnelip 192.168.0.3
1098         bridge-learning off
1099         bridge-arp-nd-suppress on
1100         bridge-unicast-flood off
1101         bridge-multicast-flood off
1102
1103
1104 auto vmbr4000
1105 iface vmbr4000 inet manual
1106         bridge_ports vxlan4000
1107         bridge_stp off
1108         bridge_fd 0
1109         hwaddress 44:39:39:FF:40:92  #must be different on each node
1110         vrf vrf1
1111 ----
1112
1113
1114 frr.conf
1115
1116 ----
1117 vrf vrf1
1118  vni 4000
1119 !
1120 router bgp 1234
1121  bgp router-id 192.168.0.3
1122  no bgp default ipv4-unicast
1123  coalesce-time 1000
1124  neighbor 192.168.0.1 remote-as 1234
1125  neighbor 192.168.0.2 remote-as 1234
1126  !
1127  address-family l2vpn evpn
1128   neighbor 192.168.0.1 activate
1129   neighbor 192.168.0.2 activate
1130   advertise-all-vni
1131  exit-address-family
1132 !
1133 router bgp 1234 vrf vrf1
1134 !
1135  bgp router-id 192.168.0.3
1136  !
1137  address-family ipv4 unicast
1138   redistribute connected
1139  exit-address-family
1140  !
1141  address-family l2vpn evpn
1142   advertise ipv4 unicast
1143  exit-address-family
1144 !
1145 line vty
1146 !
1147 ----
1148
1149 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1150 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1151 Routing to outside need the symmetric model.
1152
1153 1 gateway node
1154 ^^^^^^^^^^^^^^
1155 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1156 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1157 and announce this default gw to other proxmox nodes.
1158
1159
1160 *node1
1161
1162 ----
1163 auto vrf1
1164 iface vrf1
1165     vrf-table auto
1166
1167 auto eno1
1168 iface eno1 inet manual
1169         mtu 1550
1170
1171 auto vmbr0
1172 iface vmbr0 inet static
1173          address 192.168.0.1
1174          netmask  255.255.255.0
1175          bridge_ports eno1
1176          bridge_stp off
1177          bridge_fd 0
1178
1179 auto eno2
1180 iface eno2
1181         address 172.16.0.1
1182         netmask 255.255.255.0
1183         vrf vrf1
1184         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1185         #if you have multiple external routers, you can use ecmp balancing
1186         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1187
1188 auto vxlan2
1189 iface vxlan2 inet manual
1190         vxlan-id 2
1191         vxlan-local-tunnelip 192.168.0.1
1192         bridge-learning off
1193         bridge-arp-nd-suppress on
1194         bridge-unicast-flood off
1195         bridge-multicast-flood off
1196
1197 auto vmbr2
1198 iface vmbr2 inet static
1199         bridge_ports vxlan2
1200         bridge_stp off
1201         bridge_fd 0
1202         address 10.0.2.254
1203         netmask 255.255.255.0
1204         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1205         vrf vrf1
1206
1207 auto vxlan3
1208 iface vxlan3 inet manual
1209         vxlan-id 3
1210         vxlan-local-tunnelip 192.168.0.1
1211         bridge-learning off
1212         bridge-arp-nd-suppress on
1213         bridge-unicast-flood off
1214         bridge-multicast-flood off
1215
1216 auto vmbr3
1217 iface vmbr3 inet static
1218         bridge_ports vxlan3
1219         bridge_stp off
1220         bridge_fd 0
1221         address 10.0.3.254
1222         netmask 255.255.255.0
1223         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1224         vrf vrf1
1225
1226 #interconnect vxlan-vfr l3vni
1227 auto vxlan4000
1228 iface vxlan4000 inet manual
1229         vxlan-id 4000
1230         vxlan-local-tunnelip 192.168.0.1
1231         bridge-learning off
1232         bridge-arp-nd-suppress on
1233         bridge-unicast-flood off
1234         bridge-multicast-flood off
1235
1236 auto vmbr4000
1237 iface vmbr4000 inet manual
1238         bridge_ports vxlan4000
1239         bridge_stp off
1240         bridge_fd 0
1241         hwaddress 44:39:39:FF:40:90  #must be different on each node
1242         vrf vrf1
1243 ----
1244
1245
1246 frr.conf
1247
1248 ----
1249 vrf vrf1
1250  vni 4000
1251 !
1252 router bgp 1234
1253  bgp router-id 192.168.0.1
1254  no bgp default ipv4-unicast
1255  coalesce-time 1000
1256  neighbor 192.168.0.2 remote-as 1234
1257  neighbor 192.168.0.3 remote-as 1234
1258  !
1259  address-family l2vpn evpn
1260   neighbor 192.168.0.2 activate
1261   neighbor 192.168.0.3 activate
1262   advertise-all-vni
1263  exit-address-family
1264 !
1265 router bgp 1234 vrf vrf1
1266 !
1267  bgp router-id 172.16.0.1
1268  !
1269  address-family ipv4 unicast
1270   redistribute connected
1271   redistribute kernel !announce your default gw to all nodes
1272  exit-address-family
1273  !
1274  address-family l2vpn evpn
1275   advertise ipv4 unicast
1276  exit-address-family
1277 !
1278 line vty
1279 !
1280 ----
1281
1282
1283 * node2
1284
1285 ----
1286 auto vrf1
1287 iface vrf1
1288     vrf-table auto
1289
1290 auto eno1
1291 iface eno1 inet manual
1292         mtu 1550
1293
1294 auto vmbr0
1295 iface vmbr0 inet static
1296          address 192.168.0.2
1297          netmask  255.255.255.0
1298          bridge_ports eno1
1299          bridge_stp off
1300          bridge_fd 0
1301
1302 auto vxlan2
1303 iface vxlan2 inet manual
1304         vxlan-id 2
1305         vxlan-local-tunnelip 192.168.0.2
1306         bridge-learning off
1307         bridge-arp-nd-suppress on
1308         bridge-unicast-flood off
1309         bridge-multicast-flood off
1310
1311 auto vmbr2
1312 iface vmbr2 inet static
1313         bridge_ports vxlan2
1314         bridge_stp off
1315         bridge_fd 0
1316         address 10.0.2.254
1317         netmask 255.255.255.0
1318         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1319         vrf vrf1
1320
1321 auto vxlan3
1322 iface vxlan3 inet manual
1323         vxlan-id 3
1324         vxlan-local-tunnelip 192.168.0.2
1325         bridge-learning off
1326         bridge-arp-nd-suppress on
1327         bridge-unicast-flood off
1328         bridge-multicast-flood off
1329
1330 auto vmbr3
1331 iface vmbr3 inet static
1332         bridge_ports vxlan3
1333         bridge_stp off
1334         bridge_fd 0
1335         address 10.0.3.254
1336         netmask 255.255.255.0
1337         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1338         vrf vrf1
1339
1340 #interconnect vxlan-vfr l3vni
1341 auto vxlan4000
1342 iface vxlan4000 inet manual
1343         vxlan-id 4000
1344         vxlan-local-tunnelip 192.168.0.2
1345         bridge-learning off
1346         bridge-arp-nd-suppress on
1347         bridge-unicast-flood off
1348         bridge-multicast-flood off
1349
1350
1351 auto vmbr4000
1352 iface vmbr4000 inet manual
1353         bridge_ports vxlan4000
1354         bridge_stp off
1355         bridge_fd 0
1356         hwaddress 44:39:39:FF:40:91  #must be different on each node
1357         vrf vrf1
1358 ----
1359
1360
1361 frr.conf
1362
1363 ----
1364 vrf vrf1
1365  vni 4000
1366 !
1367 router bgp 1234
1368  bgp router-id 192.168.0.2
1369  no bgp default ipv4-unicast
1370  coalesce-time 1000
1371  neighbor 192.168.0.1 remote-as 1234
1372  neighbor 192.168.0.3 remote-as 1234
1373  !
1374  address-family l2vpn evpn
1375   neighbor 192.168.0.1 activate
1376   neighbor 192.168.0.3 activate
1377   advertise-all-vni
1378  exit-address-family
1379 !
1380 router bgp 1234 vrf vrf1
1381 !
1382  bgp router-id 192.168.0.2
1383  !
1384  address-family ipv4 unicast
1385   redistribute connected
1386  exit-address-family
1387  !
1388  address-family l2vpn evpn
1389   advertise ipv4 unicast
1390  exit-address-family
1391 !
1392 line vty
1393 !
1394 ----
1395
1396
1397 * node3
1398
1399 ----
1400 auto vrf1
1401 iface vrf1
1402     vrf-table auto
1403
1404 auto eno1
1405 iface eno1 inet manual
1406         mtu 1550
1407
1408 auto vmbr0
1409 iface vmbr0 inet static
1410          address 192.168.0.3
1411          netmask  255.255.255.0
1412          bridge_ports eno1
1413          bridge_stp off
1414          bridge_fd 0
1415
1416 auto vxlan2
1417 iface vxlan2 inet manual
1418         vxlan-id 2
1419         vxlan-local-tunnelip 192.168.0.3
1420         bridge-learning off
1421         bridge-arp-nd-suppress on
1422         bridge-unicast-flood off
1423         bridge-multicast-flood off
1424
1425 auto vmbr2
1426 iface vmbr2 inet static
1427         bridge_ports vxlan2
1428         bridge_stp off
1429         bridge_fd 0
1430         address 10.0.2.254
1431         netmask 255.255.255.0
1432         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1433         vrf vrf1
1434
1435 auto vxlan3
1436 iface vxlan3 inet manual
1437         vxlan-id 3
1438         vxlan-local-tunnelip 192.168.0.3
1439         bridge-learning off
1440         bridge-arp-nd-suppress on
1441         bridge-unicast-flood off
1442         bridge-multicast-flood off
1443
1444 auto vmbr3
1445 iface vmbr3 inet static
1446         bridge_ports vxlan3
1447         bridge_stp off
1448         bridge_fd 0
1449         address 10.0.3.254
1450         netmask 255.255.255.0
1451         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1452         vrf vrf1
1453
1454 #interconnect vxlan-vfr l3vni
1455 auto vxlan4000
1456 iface vxlan4000 inet manual
1457         vxlan-id 4000
1458         vxlan-local-tunnelip 192.168.0.3
1459         bridge-learning off
1460         bridge-arp-nd-suppress on
1461         bridge-unicast-flood off
1462         bridge-multicast-flood off
1463
1464
1465 auto vmbr4000
1466 iface vmbr4000 inet manual
1467         bridge_ports vxlan4000
1468         bridge_stp off
1469         bridge_fd 0
1470         hwaddress 44:39:39:FF:40:92  #must be different on each node
1471         vrf vrf1
1472 ----
1473
1474
1475 frr.conf
1476
1477 ----
1478 vrf vrf1
1479  vni 4000
1480 !
1481 router bgp 1234
1482  bgp router-id 192.168.0.3
1483  no bgp default ipv4-unicast
1484  coalesce-time 1000
1485  neighbor 192.168.0.1 remote-as 1234
1486  neighbor 192.168.0.2 remote-as 1234
1487  !
1488  address-family l2vpn evpn
1489   neighbor 192.168.0.1 activate
1490   neighbor 192.168.0.2 activate
1491   advertise-all-vni
1492  exit-address-family
1493 !
1494 router bgp 1234 vrf vrf1
1495 !
1496  bgp router-id 192.168.0.3
1497  !
1498  address-family ipv4 unicast
1499   redistribute connected
1500  exit-address-family
1501  !
1502  address-family l2vpn evpn
1503   advertise ipv4 unicast
1504  exit-address-family
1505 !
1506 line vty
1507 !
1508 ----
1509
1510 multiple gateway nodes
1511 ^^^^^^^^^^^^^^^^^^^^^^
1512 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1513 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1514 and announce this default gw.
1515 The external router have ecmp routes to all proxmox nodes.(balancing).
1516 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1517 vxlan the packet to final destination.
1518
1519 *node1
1520
1521 ----
1522 auto vrf1
1523 iface vrf1
1524     vrf-table auto
1525
1526 auto eno1
1527 iface eno1 inet manual
1528         mtu 1550
1529
1530 auto vmbr0
1531 iface vmbr0 inet static
1532          address 192.168.0.1
1533          netmask  255.255.255.0
1534          bridge_ports eno1
1535          bridge_stp off
1536          bridge_fd 0
1537
1538 auto eno2
1539 iface eno2
1540         address 172.16.0.1
1541         netmask 255.255.255.0
1542         vrf vrf1
1543         mtu 1550
1544         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1545         #if you have multiple external routers, you can use ecmp balancing
1546         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1547
1548 auto vxlan2
1549 iface vxlan2 inet manual
1550         vxlan-id 2
1551         vxlan-local-tunnelip 192.168.0.1
1552         bridge-learning off
1553         bridge-arp-nd-suppress on
1554         bridge-unicast-flood off
1555         bridge-multicast-flood off
1556
1557 auto vmbr2
1558 iface vmbr2 inet static
1559         bridge_ports vxlan2
1560         bridge_stp off
1561         bridge_fd 0
1562         address 10.0.2.254
1563         netmask 255.255.255.0
1564         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1565         vrf vrf1
1566
1567 auto vxlan3
1568 iface vxlan3 inet manual
1569         vxlan-id 3
1570         vxlan-local-tunnelip 192.168.0.1
1571         bridge-learning off
1572         bridge-arp-nd-suppress on
1573         bridge-unicast-flood off
1574         bridge-multicast-flood off
1575
1576 auto vmbr3
1577 iface vmbr3 inet static
1578         bridge_ports vxlan3
1579         bridge_stp off
1580         bridge_fd 0
1581         address 10.0.3.254
1582         netmask 255.255.255.0
1583         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1584         vrf vrf1
1585
1586 #interconnect vxlan-vfr l3vni
1587 auto vxlan4000
1588 iface vxlan4000 inet manual
1589         vxlan-id 4000
1590         vxlan-local-tunnelip 192.168.0.1
1591         bridge-learning off
1592         bridge-arp-nd-suppress on
1593         bridge-unicast-flood off
1594         bridge-multicast-flood off
1595
1596 auto vmbr4000
1597 iface vmbr4000 inet manual
1598         bridge_ports vxlan4000
1599         bridge_stp off
1600         bridge_fd 0
1601         hwaddress 44:39:39:FF:40:90  #must be different on each node
1602         vrf vrf1
1603 ----
1604
1605
1606 frr.conf
1607
1608 ----
1609 vrf vrf1
1610  vni 4000
1611 !
1612 router bgp 1234
1613  bgp router-id 192.168.0.1
1614  no bgp default ipv4-unicast
1615  coalesce-time 1000
1616  neighbor 192.168.0.2 remote-as 1234
1617  neighbor 192.168.0.3 remote-as 1234
1618  !
1619  address-family l2vpn evpn
1620   neighbor 192.168.0.2 activate
1621   neighbor 192.168.0.3 activate
1622   advertise-all-vni
1623  exit-address-family
1624 !
1625 router bgp 1234 vrf vrf1
1626 !
1627  bgp router-id 172.16.0.1
1628  !
1629  address-family ipv4 unicast
1630   redistribute connected
1631   redistribute kernel !announce your default gw to all nodes
1632  exit-address-family
1633  !
1634  address-family l2vpn evpn
1635   advertise ipv4 unicast
1636  exit-address-family
1637 !
1638 line vty
1639 !
1640 ----
1641
1642
1643 * node2
1644
1645 ----
1646 auto vrf1
1647 iface vrf1
1648     vrf-table auto
1649
1650 auto eno1
1651 iface eno1 inet manual
1652         mtu 1550
1653
1654 auto vmbr0
1655 iface vmbr0 inet static
1656          address 192.168.0.2
1657          netmask  255.255.255.0
1658          bridge_ports eno1
1659          bridge_stp off
1660          bridge_fd 0
1661
1662 auto eno2
1663 iface eno2
1664         address 172.16.0.3
1665         netmask 255.255.255.0
1666         vrf vrf1
1667         mtu 1550
1668         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1669         #if you have multiple external routers, you can use ecmp balancing
1670         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1671
1672 auto vxlan2
1673 iface vxlan2 inet manual
1674         vxlan-id 2
1675         vxlan-local-tunnelip 192.168.0.2
1676         bridge-learning off
1677         bridge-arp-nd-suppress on
1678         bridge-unicast-flood off
1679         bridge-multicast-flood off
1680
1681 auto vmbr2
1682 iface vmbr2 inet static
1683         bridge_ports vxlan2
1684         bridge_stp off
1685         bridge_fd 0
1686         address 10.0.2.254
1687         netmask 255.255.255.0
1688         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1689         vrf vrf1
1690
1691 auto vxlan3
1692 iface vxlan3 inet manual
1693         vxlan-id 3
1694         vxlan-local-tunnelip 192.168.0.2
1695         bridge-learning off
1696         bridge-arp-nd-suppress on
1697         bridge-unicast-flood off
1698         bridge-multicast-flood off
1699
1700 auto vmbr3
1701 iface vmbr3 inet static
1702         bridge_ports vxlan3
1703         bridge_stp off
1704         bridge_fd 0
1705         address 10.0.3.254
1706         netmask 255.255.255.0
1707         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1708         vrf vrf1
1709
1710 #interconnect vxlan-vfr l3vni
1711 auto vxlan4000
1712 iface vxlan4000 inet manual
1713         vxlan-id 4000
1714         vxlan-local-tunnelip 192.168.0.2
1715         bridge-learning off
1716         bridge-arp-nd-suppress on
1717         bridge-unicast-flood off
1718         bridge-multicast-flood off
1719
1720
1721 auto vmbr4000
1722 iface vmbr4000 inet manual
1723         bridge_ports vxlan4000
1724         bridge_stp off
1725         bridge_fd 0
1726         hwaddress 44:39:39:FF:40:91  #must be different on each node
1727         vrf vrf1
1728 ----
1729
1730
1731 frr.conf
1732
1733 ----
1734 vrf vrf1
1735  vni 4000
1736 !
1737 router bgp 1234
1738  bgp router-id 192.168.0.2
1739  no bgp default ipv4-unicast
1740  coalesce-time 1000
1741  neighbor 192.168.0.1 remote-as 1234
1742  neighbor 192.168.0.3 remote-as 1234
1743  !
1744  address-family l2vpn evpn
1745   neighbor 192.168.0.1 activate
1746   neighbor 192.168.0.3 activate
1747   advertise-all-vni
1748  exit-address-family
1749 !
1750 router bgp 1234 vrf vrf1
1751 !
1752  bgp router-id 172.16.0.2
1753  !
1754  address-family ipv4 unicast
1755   redistribute connected
1756   redistribute kernel !announce your default gw to all nodes
1757  exit-address-family
1758  !
1759  address-family l2vpn evpn
1760   advertise ipv4 unicast
1761  exit-address-family
1762 !
1763 line vty
1764 !
1765 ----
1766
1767
1768 * node3
1769
1770 ----
1771 auto vrf1
1772 iface vrf1
1773     vrf-table auto
1774
1775 auto eno1
1776 iface eno1 inet manual
1777         mtu 1550
1778
1779 auto vmbr0
1780 iface vmbr0 inet static
1781          address 192.168.0.3
1782          netmask  255.255.255.0
1783          bridge_ports eno1
1784          bridge_stp off
1785          bridge_fd 0
1786
1787 auto eno2
1788 iface eno2
1789         address 172.16.0.3
1790         netmask 255.255.255.0
1791         vrf vrf1
1792         mtu 1550
1793         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1794         #if you have multiple external routers, you can use ecmp balancing
1795         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1796
1797 auto vxlan2
1798 iface vxlan2 inet manual
1799         vxlan-id 2
1800         vxlan-local-tunnelip 192.168.0.3
1801         bridge-learning off
1802         bridge-arp-nd-suppress on
1803         bridge-unicast-flood off
1804         bridge-multicast-flood off
1805
1806 auto vmbr2
1807 iface vmbr2 inet static
1808         bridge_ports vxlan2
1809         bridge_stp off
1810         bridge_fd 0
1811         address 10.0.2.254
1812         netmask 255.255.255.0
1813         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1814         vrf vrf1
1815
1816 auto vxlan3
1817 iface vxlan3 inet manual
1818         vxlan-id 3
1819         vxlan-local-tunnelip 192.168.0.3
1820         bridge-learning off
1821         bridge-arp-nd-suppress on
1822         bridge-unicast-flood off
1823         bridge-multicast-flood off
1824
1825 auto vmbr3
1826 iface vmbr3 inet static
1827         bridge_ports vxlan3
1828         bridge_stp off
1829         bridge_fd 0
1830         address 10.0.3.254
1831         netmask 255.255.255.0
1832         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1833         vrf vrf1
1834
1835 #interconnect vxlan-vfr l3vni
1836 auto vxlan4000
1837 iface vxlan4000 inet manual
1838         vxlan-id 4000
1839         vxlan-local-tunnelip 192.168.0.3
1840         bridge-learning off
1841         bridge-arp-nd-suppress on
1842         bridge-unicast-flood off
1843         bridge-multicast-flood off
1844
1845
1846 auto vmbr4000
1847 iface vmbr4000 inet manual
1848         bridge_ports vxlan4000
1849         bridge_stp off
1850         bridge_fd 0
1851         hwaddress 44:39:39:FF:40:92  #must be different on each node
1852         vrf vrf1
1853 ----
1854
1855
1856 frr.conf
1857
1858 ----
1859 vrf vrf1
1860  vni 4000
1861 !
1862 router bgp 1234
1863  bgp router-id 192.168.0.3
1864  no bgp default ipv4-unicast
1865  coalesce-time 1000
1866  neighbor 192.168.0.1 remote-as 1234
1867  neighbor 192.168.0.2 remote-as 1234
1868  !
1869  address-family l2vpn evpn
1870   neighbor 192.168.0.1 activate
1871   neighbor 192.168.0.2 activate
1872   advertise-all-vni
1873  exit-address-family
1874 !
1875 router bgp 1234 vrf vrf1
1876 !
1877  bgp router-id 172.16.0.3
1878  !
1879  address-family ipv4 unicast
1880   redistribute connected
1881   redistribute kernel !announce your default gw to all nodes
1882  exit-address-family
1883  !
1884  address-family l2vpn evpn
1885   advertise ipv4 unicast
1886  exit-address-family
1887 !
1888 line vty
1889 !
1890 ----
1891
1892 Note
1893 ^^^^
1894
1895 If your external router don't support ecmp to reach multiple proxmox nodes,
1896 you can setup an HA floating vip on proxmox nodes with vrrp
1897
1898 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1899 Node1 is the primary and failover to node2 in case of failure.
1900
1901
1902 * node1
1903
1904 ----
1905 auto eno2
1906 iface eno2
1907         address 172.16.0.1
1908         netmask 255.255.255.0
1909         vrf vrf1
1910         mtu 1550
1911         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1912         vrrp-id 1
1913         vrrp-priority 1
1914         vrrp-virtual-ip 172.16.0.10
1915 ----
1916
1917 * node2
1918
1919 ----
1920 auto eno2
1921 iface eno2
1922         address 172.16.0.2
1923         netmask 255.255.255.0
1924         mtu 1550
1925         vrf vrf1
1926         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1927         vrrp-id 1
1928         vrrp-priority 2
1929         vrrp-virtual-ip 172.16.0.10
1930 ----
1931
1932