vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  coalesce-time 1000
 371  neighbor 192.168.0.2 remote-as 1234
 372  neighbor 192.168.0.3 remote-as 1234
 373  !
 374  address-family l2vpn evpn
 375   neighbor 192.168.0.2 activate
 376   neighbor 192.168.0.3 activate
 377   advertise-all-vni
 378  exit-address-family
 379 !
 380 line vty
 381 !
 382 ----
 383
 384
 385 * node2
 386
 387 ----
 388 auto eno1
 389 iface eno1 inet manual
 390         mtu 1550
 391
 392 auto vmbr0
 393 iface vmbr0 inet static
 394         address 192.168.0.2
 395         netmask 255.255.255.0
 396         bridge_ports eno1
 397         bridge_stp off
 398         bridge_fd 0
 399
 400 auto vxlan2
 401 iface vxlan2 inet manual
 402         vxlan-id 2
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr2
 411 iface vmbr2 inet manual
 412         bridge_ports vxlan2
 413         bridge_stp off
 414         bridge_fd 0
 415
 416 auto vxlan3
 417 iface vxlan3 inet manual
 418         vxlan-id 3
 419         vxlan-local-tunnelip 192.168.0.2
 420         bridge-learning off
 421         bridge-arp-nd-suppress on
 422         bridge-unicast-flood off
 423         bridge-multicast-flood off
 424
 425
 426 auto vmbr3
 427 iface vmbr3 inet manual
 428         bridge_ports vxlan3
 429         bridge_stp off
 430         bridge_fd 0
 431 ----
 432
 433
 434 /etc/frr/frr.conf
 435
 436 ----
 437 router bgp 1234
 438  no bgp default ipv4-unicast
 439  coalesce-time 1000
 440  neighbor 192.168.0.1 remote-as 1234
 441  neighbor 192.168.0.3 remote-as 1234
 442  !
 443  address-family l2vpn evpn
 444   neighbor 192.168.0.1 activate
 445   neighbor 192.168.0.3 activate
 446   advertise-all-vni
 447  exit-address-family
 448 !
 449 line vty
 450 !
 451 ----
 452
 453
 454 * node3
 455
 456 ----
 457 auto eno1
 458 iface eno1 inet manual
 459         mtu 1550
 460
 461 auto vmbr0
 462 iface vmbr0 inet static
 463         address 192.168.0.2
 464         netmask 255.255.255.0
 465         bridge_ports eno1
 466         bridge_stp off
 467         bridge_fd 0
 468
 469 auto vxlan2
 470 iface vxlan2 inet manual
 471         vxlan-id 2
 472         vxlan-local-tunnelip 192.168.0.3
 473         bridge-learning off
 474         bridge-arp-nd-suppress on
 475         bridge-unicast-flood off
 476         bridge-multicast-flood off
 477
 478
 479 auto vmbr2
 480 iface vmbr2 inet manual
 481         bridge_ports vxlan2
 482         bridge_stp off
 483         bridge_fd 0
 484
 485 auto vxlan3
 486 iface vxlan3 inet manual
 487         vxlan-id 3
 488         vxlan-local-tunnelip 192.168.0.3
 489         bridge-learning off
 490         bridge-arp-nd-suppress on
 491         bridge-unicast-flood off
 492         bridge-multicast-flood off
 493
 494
 495 auto vmbr3
 496 iface vmbr3 inet manual
 497         bridge_ports vxlan3
 498         bridge_stp off
 499         bridge_fd 0
 500 ----
 501
 502
 503 /etc/frr/frr.conf
 504
 505
 506 ----
 507 router bgp 1234
 508  no bgp default ipv4-unicast
 509  coalesce-time 1000
 510  neighbor 192.168.0.1 remote-as 1234
 511  neighbor 192.168.0.2 remote-as 1234
 512  !
 513  address-family l2vpn evpn
 514   neighbor 192.168.0.1 activate
 515   neighbor 192.168.0.2 activate
 516   advertise-all-vni
 517  exit-address-family
 518 !
 519 line vty
 520 !
 521 ----
 522
 523 VXLAN layer3 routing with anycast gateway
 524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 525
 526 With this need, each vmbr bridge will be the gateway for the vm.
 527 Same vmbr on different node, will have same ip address and same mac address,
 528 to have working vm live migration and no network disruption.
 529
 530 VXLAN layer3 routing only work with FRR and non-aware bridge.
 531 (vlan aware bridge support is buggy currently).
 532
 533 asymmetric model
 534 ^^^^^^^^^^^^^^^^
 535
 536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 537
 538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 539 but only bridging on the egress.
 540 This results in bi-directional VXLAN traffic traveling on different VNIs
 541 in each direction (always the destination VNI) across the routed infrastructure.
 542
 543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 544
 545
 546 sysctl.conf tuning
 547
 548 ----
 549 #enable routing
 550 net.ipv4.ip_forward=1
 551 net.ipv6.conf.all.forwarding=1
 552 ----
 553
 554 * node1
 555
 556 ----
 557 auto eno1
 558 iface eno1 inet manual
 559         mtu 1550
 560
 561 auto vmbr0
 562 iface vmbr0 inet static
 563          address 192.168.0.1
 564          netmask  255.255.255.0
 565          bridge_ports eno1
 566          bridge_stp off
 567          bridge_fd 0
 568
 569 auto vxlan2
 570 iface vxlan2 inet manual
 571         vxlan-id 2
 572         vxlan-local-tunnelip 192.168.0.1
 573         bridge-learning off
 574         bridge-arp-nd-suppress on
 575         bridge-unicast-flood off
 576         bridge-multicast-flood off
 577
 578
 579 auto vmbr2
 580 iface vmbr2 inet static
 581         address 10.0.2.254
 582         netmask 255.255.255.0
 583         hwaddress 44:39:39:FF:40:94
 584         bridge_ports vxlan2
 585         bridge_stp off
 586         bridge_fd 0
 587
 588
 589 auto vxlan3
 590 iface vxlan3 inet manual
 591         vxlan-id 3
 592         vxlan-local-tunnelip 192.168.0.1
 593         bridge-learning off
 594         bridge-arp-nd-suppress on
 595         bridge-unicast-flood off
 596         bridge-multicast-flood off
 597
 598
 599 auto vmbr3
 600 iface vmbr3 inet static
 601         address 10.0.3.254
 602         netmask 255.255.255.0
 603         hwaddress 44:39:39:FF:40:94
 604         bridge_ports vxlan3
 605         bridge_stp off
 606         bridge_fd 0
 607 ----
 608
 609
 610 frr.conf
 611
 612 ----
 613 router bgp 1234
 614  bgp router-id 192.168.0.1
 615  no bgp default ipv4-unicast
 616  coalesce-time 1000
 617  neighbor 192.168.0.2 remote-as 1234
 618  neighbor 192.168.0.3 remote-as 1234
 619  !
 620  address-family l2vpn evpn
 621   neighbor 192.168.0.2 activate
 622   neighbor 192.168.0.3 activate
 623   advertise-all-vni
 624  exit-address-family
 625 !
 626 line vty
 627 !
 628 ----
 629
 630
 631 * node2
 632
 633 ----
 634 auto eno1
 635 iface eno1 inet manual
 636         mtu 1550
 637
 638 auto vmbr0
 639 iface vmbr0 inet static
 640          address 192.168.0.2
 641          netmask  255.255.255.0
 642          bridge_ports eno1
 643          bridge_stp off
 644          bridge_fd 0
 645
 646 auto vxlan2
 647 iface vxlan2 inet manual
 648         vxlan-id 2
 649         vxlan-local-tunnelip 192.168.0.2
 650         bridge-learning off
 651         bridge-arp-nd-suppress on
 652         bridge-unicast-flood off
 653         bridge-multicast-flood off
 654
 655
 656 auto vmbr2
 657 iface vmbr2 inet static
 658         address 10.0.2.254
 659         netmask 255.255.255.0
 660         hwaddress 44:39:39:FF:40:94
 661         bridge_ports vxlan2
 662         bridge_stp off
 663         bridge_fd 0
 664
 665
 666 auto vxlan3
 667 iface vxlan3 inet manual
 668         vxlan-id 3
 669         vxlan-local-tunnelip 192.168.0.2
 670         bridge-learning off
 671         bridge-arp-nd-suppress on
 672         bridge-unicast-flood off
 673         bridge-multicast-flood off
 674
 675
 676 auto vmbr3
 677 iface vmbr3 inet static
 678         address 10.0.3.254
 679         netmask 255.255.255.0
 680         hwaddress 44:39:39:FF:40:94
 681         bridge_ports vxlan3
 682         bridge_stp off
 683         bridge_fd 0
 684 ----
 685
 686
 687 frr.conf
 688
 689 ----
 690 router bgp 1234
 691  bgp router-id 192.168.0.2
 692  no bgp default ipv4-unicast
 693  coalesce-time 1000
 694  neighbor 192.168.0.1 remote-as 1234
 695  neighbor 192.168.0.3 remote-as 1234
 696  !
 697  address-family l2vpn evpn
 698   neighbor 192.168.0.1 activate
 699   neighbor 192.168.0.3 activate
 700   advertise-all-vni
 701  exit-address-family
 702 !
 703 line vty
 704 !
 705 ----
 706
 707
 708 * node3
 709
 710 ----
 711 auto eno1
 712 iface eno1 inet manual
 713         mtu 1550
 714
 715 auto vmbr0
 716 iface vmbr0 inet static
 717          address 192.168.0.3
 718          netmask  255.255.255.0
 719          bridge_ports eno1
 720          bridge_stp off
 721          bridge_fd 0
 722
 723 auto vxlan2
 724 iface vxlan2 inet manual
 725         vxlan-id 2
 726         vxlan-local-tunnelip 192.168.0.3
 727         bridge-learning off
 728         bridge-arp-nd-suppress on
 729         bridge-unicast-flood off
 730         bridge-multicast-flood off
 731
 732
 733 auto vmbr2
 734 iface vmbr2 inet static
 735         address 10.0.2.254
 736         netmask 255.255.255.0
 737         hwaddress 44:39:39:FF:40:94
 738         bridge_ports vxlan2
 739         bridge_stp off
 740         bridge_fd 0
 741
 742
 743 auto vxlan3
 744 iface vxlan3 inet manual
 745         vxlan-id 3
 746         vxlan-local-tunnelip 192.168.0.3
 747         bridge-learning off
 748         bridge-arp-nd-suppress on
 749         bridge-unicast-flood off
 750         bridge-multicast-flood off
 751
 752
 753 auto vmbr3
 754 iface vmbr3 inet static
 755         address 10.0.3.254
 756         netmask 255.255.255.0
 757         hwaddress 44:39:39:FF:40:94
 758         bridge_ports vxlan3
 759         bridge_stp off
 760         bridge_fd 0
 761 ----
 762
 763
 764 frr.conf
 765
 766 ----
 767 router bgp 1234
 768  bgp router-id 192.168.0.3
 769  no bgp default ipv4-unicast
 770  coalesce-time 1000
 771  neighbor 192.168.0.1 remote-as 1234
 772  neighbor 192.168.0.2 remote-as 1234
 773  !
 774  address-family l2vpn evpn
 775   neighbor 192.168.0.1 activate
 776   neighbor 192.168.0.2 activate
 777   advertise-all-vni
 778  exit-address-family
 779 !
 780 line vty
 781 !
 782 ----
 783
 784
 785 symmetric model
 786 ^^^^^^^^^^^^^^^
 787
 788 With this model, you don't need to have all vxlan on all nodes.
 789 This model will also be needed to route traffic to an external router.
 790
 791 The symmetric model routes and bridges on both the ingress and the egress leafs.
 792 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 793 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 794 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 795 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 796
 797 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 798
 799 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 800
 801 sysctl.conf tuning
 802
 803 ----
 804 #enable routing
 805 net.ipv4.ip_forward=1
 806 net.ipv6.conf.all.forwarding=1
 807 #disable reverse path filtering
 808 net.ipv4.conf.default.rp_filter=0
 809 net.ipv4.conf.all.rp_filter=0
 810 ----
 811
 812 * node1
 813
 814 ----
 815 auto vrf1
 816 iface vrf1
 817     vrf-table auto
 818
 819 auto eno1
 820 iface eno1 inet manual
 821         mtu 1550
 822
 823 auto vmbr0
 824 iface vmbr0 inet static
 825          address 192.168.0.1
 826          netmask  255.255.255.0
 827          bridge_ports eno1
 828          bridge_stp off
 829          bridge_fd 0
 830
 831 auto vxlan2
 832 iface vxlan2 inet manual
 833         vxlan-id 2
 834         vxlan-local-tunnelip 192.168.0.1
 835         bridge-learning off
 836         bridge-arp-nd-suppress on
 837         bridge-unicast-flood off
 838         bridge-multicast-flood off
 839
 840 auto vmbr2
 841 iface vmbr2 inet static
 842         bridge_ports vxlan2
 843         bridge_stp off
 844         bridge_fd 0
 845         address 10.0.2.254
 846         netmask 255.255.255.0
 847         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 848         vrf vrf1
 849
 850 auto vxlan3
 851 iface vxlan3 inet manual
 852         vxlan-id 3
 853         vxlan-local-tunnelip 192.168.0.1
 854         bridge-learning off
 855         bridge-arp-nd-suppress on
 856         bridge-unicast-flood off
 857         bridge-multicast-flood off
 858
 859 auto vmbr3
 860 iface vmbr3 inet static
 861         bridge_ports vxlan3
 862         bridge_stp off
 863         bridge_fd 0
 864         address 10.0.3.254
 865         netmask 255.255.255.0
 866         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 867         vrf vrf1
 868
 869 #interconnect vxlan-vfr l3vni
 870 auto vxlan4000
 871 iface vxlan4000 inet manual
 872         vxlan-id 4000
 873         vxlan-local-tunnelip 192.168.0.1
 874         bridge-learning off
 875         bridge-arp-nd-suppress on
 876         bridge-unicast-flood off
 877         bridge-multicast-flood off
 878
 879
 880 auto vmbr4000
 881 iface vmbr4000 inet manual
 882         bridge_ports vxlan4000
 883         bridge_stp off
 884         bridge_fd 0
 885         hwaddress 44:39:39:FF:40:90  #must be different on each node
 886         vrf vrf1
 887 ----
 888
 889 frr.conf
 890
 891 ----
 892 vrf vrf1
 893  vni 4000
 894 !
 895 router bgp 1234
 896  bgp router-id 192.168.0.1
 897  no bgp default ipv4-unicast
 898  coalesce-time 1000
 899  neighbor 192.168.0.2 remote-as 1234
 900  neighbor 192.168.0.3 remote-as 1234
 901  !
 902  address-family l2vpn evpn
 903   neighbor 192.168.0.2 activate
 904   neighbor 192.168.0.3 activate
 905   advertise-all-vni
 906  exit-address-family
 907 !
 908 router bgp 1234 vrf vrf1
 909 !
 910  bgp router-id 192.168.0.1
 911  !
 912  address-family ipv4 unicast
 913   redistribute connected
 914  exit-address-family
 915  !
 916  address-family l2vpn evpn
 917   advertise ipv4 unicast
 918  exit-address-family
 919 !
 920 line vty
 921 !
 922 ----
 923
 924
 925 * node2
 926
 927 ----
 928 auto vrf1
 929 iface vrf1
 930     vrf-table auto
 931
 932 auto eno1
 933 iface eno1 inet manual
 934         mtu 1550
 935
 936 auto vmbr0
 937 iface vmbr0 inet static
 938          address 192.168.0.2
 939          netmask  255.255.255.0
 940          bridge_ports eno1
 941          bridge_stp off
 942          bridge_fd 0
 943
 944 auto vxlan2
 945 iface vxlan2 inet manual
 946         vxlan-id 2
 947         vxlan-local-tunnelip 192.168.0.2
 948         bridge-learning off
 949         bridge-arp-nd-suppress on
 950         bridge-unicast-flood off
 951         bridge-multicast-flood off
 952
 953 auto vmbr2
 954 iface vmbr2 inet static
 955         bridge_ports vxlan2
 956         bridge_stp off
 957         bridge_fd 0
 958         address 10.0.2.254
 959         netmask 255.255.255.0
 960         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 961         vrf vrf1
 962
 963 auto vxlan3
 964 iface vxlan3 inet manual
 965         vxlan-id 3
 966         vxlan-local-tunnelip 192.168.0.2
 967         bridge-learning off
 968         bridge-arp-nd-suppress on
 969         bridge-unicast-flood off
 970         bridge-multicast-flood off
 971
 972 auto vmbr3
 973 iface vmbr3 inet static
 974         bridge_ports vxlan3
 975         bridge_stp off
 976         bridge_fd 0
 977         address 10.0.3.254
 978         netmask 255.255.255.0
 979         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 980         vrf vrf1
 981
 982 #interconnect vxlan-vfr l3vni
 983 auto vxlan4000
 984 iface vxlan4000 inet manual
 985         vxlan-id 4000
 986         vxlan-local-tunnelip 192.168.0.2
 987         bridge-learning off
 988         bridge-arp-nd-suppress on
 989         bridge-unicast-flood off
 990         bridge-multicast-flood off
 991
 992
 993 auto vmbr4000
 994 iface vmbr4000 inet manual
 995         bridge_ports vxlan4000
 996         bridge_stp off
 997         bridge_fd 0
 998         hwaddress 44:39:39:FF:40:91  #must be different on each node
 999         vrf vrf1
1000 ----
1001
1002
1003 frr.conf
1004
1005 ----
1006 vrf vrf1
1007  vni 4000
1008 !
1009 router bgp 1234
1010  bgp router-id 192.168.0.2
1011  no bgp default ipv4-unicast
1012  coalesce-time 1000
1013  neighbor 192.168.0.1 remote-as 1234
1014  neighbor 192.168.0.3 remote-as 1234
1015  !
1016  address-family l2vpn evpn
1017   neighbor 192.168.0.1 activate
1018   neighbor 192.168.0.3 activate
1019   advertise-all-vni
1020  exit-address-family
1021 !
1022 router bgp 1234 vrf vrf1
1023 !
1024  bgp router-id 192.168.0.2
1025  !
1026  address-family ipv4 unicast
1027   redistribute connected
1028  exit-address-family
1029  !
1030  address-family l2vpn evpn
1031   advertise ipv4 unicast
1032  exit-address-family
1033 !
1034 line vty
1035 !
1036 ----
1037
1038
1039 * node3
1040
1041 ----
1042 auto vrf1
1043 iface vrf1
1044     vrf-table auto
1045
1046 auto eno1
1047 iface eno1 inet manual
1048         mtu 1550
1049
1050 auto vmbr0
1051 iface vmbr0 inet static
1052          address 192.168.0.3
1053          netmask  255.255.255.0
1054          bridge_ports eno1
1055          bridge_stp off
1056          bridge_fd 0
1057
1058 auto vxlan2
1059 iface vxlan2 inet manual
1060         vxlan-id 2
1061         vxlan-local-tunnelip 192.168.0.3
1062         bridge-learning off
1063         bridge-arp-nd-suppress on
1064         bridge-unicast-flood off
1065         bridge-multicast-flood off
1066
1067 auto vmbr2
1068 iface vmbr2 inet static
1069         bridge_ports vxlan2
1070         bridge_stp off
1071         bridge_fd 0
1072         address 10.0.2.254
1073         netmask 255.255.255.0
1074         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1075         vrf vrf1
1076
1077 auto vxlan3
1078 iface vxlan3 inet manual
1079         vxlan-id 3
1080         vxlan-local-tunnelip 192.168.0.3
1081         bridge-learning off
1082         bridge-arp-nd-suppress on
1083         bridge-unicast-flood off
1084         bridge-multicast-flood off
1085
1086 auto vmbr3
1087 iface vmbr3 inet static
1088         bridge_ports vxlan3
1089         bridge_stp off
1090         bridge_fd 0
1091         address 10.0.3.254
1092         netmask 255.255.255.0
1093         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1094         vrf vrf1
1095
1096 #interconnect vxlan-vfr l3vni
1097 auto vxlan4000
1098 iface vxlan4000 inet manual
1099         vxlan-id 4000
1100         vxlan-local-tunnelip 192.168.0.3
1101         bridge-learning off
1102         bridge-arp-nd-suppress on
1103         bridge-unicast-flood off
1104         bridge-multicast-flood off
1105
1106
1107 auto vmbr4000
1108 iface vmbr4000 inet manual
1109         bridge_ports vxlan4000
1110         bridge_stp off
1111         bridge_fd 0
1112         hwaddress 44:39:39:FF:40:92  #must be different on each node
1113         vrf vrf1
1114 ----
1115
1116
1117 frr.conf
1118
1119 ----
1120 vrf vrf1
1121  vni 4000
1122 !
1123 router bgp 1234
1124  bgp router-id 192.168.0.3
1125  no bgp default ipv4-unicast
1126  coalesce-time 1000
1127  neighbor 192.168.0.1 remote-as 1234
1128  neighbor 192.168.0.2 remote-as 1234
1129  !
1130  address-family l2vpn evpn
1131   neighbor 192.168.0.1 activate
1132   neighbor 192.168.0.2 activate
1133   advertise-all-vni
1134  exit-address-family
1135 !
1136 router bgp 1234 vrf vrf1
1137 !
1138  bgp router-id 192.168.0.3
1139  !
1140  address-family ipv4 unicast
1141   redistribute connected
1142  exit-address-family
1143  !
1144  address-family l2vpn evpn
1145   advertise ipv4 unicast
1146  exit-address-family
1147 !
1148 line vty
1149 !
1150 ----
1151
1152 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1153 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1154 Routing to outside need the symmetric model.
1155
1156 1 gateway node
1157 ^^^^^^^^^^^^^^
1158 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1159 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1160 and announce this default gw to other proxmox nodes.
1161
1162
1163 *node1
1164
1165 ----
1166 auto vrf1
1167 iface vrf1
1168     vrf-table auto
1169
1170 auto eno1
1171 iface eno1 inet manual
1172         mtu 1550
1173
1174 auto vmbr0
1175 iface vmbr0 inet static
1176          address 192.168.0.1
1177          netmask  255.255.255.0
1178          bridge_ports eno1
1179          bridge_stp off
1180          bridge_fd 0
1181
1182 auto eno2
1183 iface eno2
1184         address 172.16.0.1
1185         netmask 255.255.255.0
1186         vrf vrf1
1187         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1188         #if you have multiple external routers, you can use ecmp balancing
1189         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1190
1191 auto vxlan2
1192 iface vxlan2 inet manual
1193         vxlan-id 2
1194         vxlan-local-tunnelip 192.168.0.1
1195         bridge-learning off
1196         bridge-arp-nd-suppress on
1197         bridge-unicast-flood off
1198         bridge-multicast-flood off
1199
1200 auto vmbr2
1201 iface vmbr2 inet static
1202         bridge_ports vxlan2
1203         bridge_stp off
1204         bridge_fd 0
1205         address 10.0.2.254
1206         netmask 255.255.255.0
1207         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1208         vrf vrf1
1209
1210 auto vxlan3
1211 iface vxlan3 inet manual
1212         vxlan-id 3
1213         vxlan-local-tunnelip 192.168.0.1
1214         bridge-learning off
1215         bridge-arp-nd-suppress on
1216         bridge-unicast-flood off
1217         bridge-multicast-flood off
1218
1219 auto vmbr3
1220 iface vmbr3 inet static
1221         bridge_ports vxlan3
1222         bridge_stp off
1223         bridge_fd 0
1224         address 10.0.3.254
1225         netmask 255.255.255.0
1226         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1227         vrf vrf1
1228
1229 #interconnect vxlan-vfr l3vni
1230 auto vxlan4000
1231 iface vxlan4000 inet manual
1232         vxlan-id 4000
1233         vxlan-local-tunnelip 192.168.0.1
1234         bridge-learning off
1235         bridge-arp-nd-suppress on
1236         bridge-unicast-flood off
1237         bridge-multicast-flood off
1238
1239 auto vmbr4000
1240 iface vmbr4000 inet manual
1241         bridge_ports vxlan4000
1242         bridge_stp off
1243         bridge_fd 0
1244         hwaddress 44:39:39:FF:40:90  #must be different on each node
1245         vrf vrf1
1246 ----
1247
1248
1249 frr.conf
1250
1251 ----
1252 vrf vrf1
1253  vni 4000
1254 !
1255 router bgp 1234
1256  bgp router-id 192.168.0.1
1257  no bgp default ipv4-unicast
1258  coalesce-time 1000
1259  neighbor 192.168.0.2 remote-as 1234
1260  neighbor 192.168.0.3 remote-as 1234
1261  !
1262  address-family l2vpn evpn
1263   neighbor 192.168.0.2 activate
1264   neighbor 192.168.0.3 activate
1265   advertise-all-vni
1266  exit-address-family
1267 !
1268 router bgp 1234 vrf vrf1
1269 !
1270  bgp router-id 172.16.0.1
1271  !
1272  address-family ipv4 unicast
1273   redistribute connected
1274   redistribute kernel !announce your default gw to all nodes
1275  exit-address-family
1276  !
1277  address-family l2vpn evpn
1278   advertise ipv4 unicast
1279  exit-address-family
1280 !
1281 line vty
1282 !
1283 ----
1284
1285
1286 * node2
1287
1288 ----
1289 auto vrf1
1290 iface vrf1
1291     vrf-table auto
1292
1293 auto eno1
1294 iface eno1 inet manual
1295         mtu 1550
1296
1297 auto vmbr0
1298 iface vmbr0 inet static
1299          address 192.168.0.2
1300          netmask  255.255.255.0
1301          bridge_ports eno1
1302          bridge_stp off
1303          bridge_fd 0
1304
1305 auto vxlan2
1306 iface vxlan2 inet manual
1307         vxlan-id 2
1308         vxlan-local-tunnelip 192.168.0.2
1309         bridge-learning off
1310         bridge-arp-nd-suppress on
1311         bridge-unicast-flood off
1312         bridge-multicast-flood off
1313
1314 auto vmbr2
1315 iface vmbr2 inet static
1316         bridge_ports vxlan2
1317         bridge_stp off
1318         bridge_fd 0
1319         address 10.0.2.254
1320         netmask 255.255.255.0
1321         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1322         vrf vrf1
1323
1324 auto vxlan3
1325 iface vxlan3 inet manual
1326         vxlan-id 3
1327         vxlan-local-tunnelip 192.168.0.2
1328         bridge-learning off
1329         bridge-arp-nd-suppress on
1330         bridge-unicast-flood off
1331         bridge-multicast-flood off
1332
1333 auto vmbr3
1334 iface vmbr3 inet static
1335         bridge_ports vxlan3
1336         bridge_stp off
1337         bridge_fd 0
1338         address 10.0.3.254
1339         netmask 255.255.255.0
1340         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1341         vrf vrf1
1342
1343 #interconnect vxlan-vfr l3vni
1344 auto vxlan4000
1345 iface vxlan4000 inet manual
1346         vxlan-id 4000
1347         vxlan-local-tunnelip 192.168.0.2
1348         bridge-learning off
1349         bridge-arp-nd-suppress on
1350         bridge-unicast-flood off
1351         bridge-multicast-flood off
1352
1353
1354 auto vmbr4000
1355 iface vmbr4000 inet manual
1356         bridge_ports vxlan4000
1357         bridge_stp off
1358         bridge_fd 0
1359         hwaddress 44:39:39:FF:40:91  #must be different on each node
1360         vrf vrf1
1361 ----
1362
1363
1364 frr.conf
1365
1366 ----
1367 vrf vrf1
1368  vni 4000
1369 !
1370 router bgp 1234
1371  bgp router-id 192.168.0.2
1372  no bgp default ipv4-unicast
1373  coalesce-time 1000
1374  neighbor 192.168.0.1 remote-as 1234
1375  neighbor 192.168.0.3 remote-as 1234
1376  !
1377  address-family l2vpn evpn
1378   neighbor 192.168.0.1 activate
1379   neighbor 192.168.0.3 activate
1380   advertise-all-vni
1381  exit-address-family
1382 !
1383 router bgp 1234 vrf vrf1
1384 !
1385  bgp router-id 192.168.0.2
1386  !
1387  address-family ipv4 unicast
1388   redistribute connected
1389  exit-address-family
1390  !
1391  address-family l2vpn evpn
1392   advertise ipv4 unicast
1393  exit-address-family
1394 !
1395 line vty
1396 !
1397 ----
1398
1399
1400 * node3
1401
1402 ----
1403 auto vrf1
1404 iface vrf1
1405     vrf-table auto
1406
1407 auto eno1
1408 iface eno1 inet manual
1409         mtu 1550
1410
1411 auto vmbr0
1412 iface vmbr0 inet static
1413          address 192.168.0.3
1414          netmask  255.255.255.0
1415          bridge_ports eno1
1416          bridge_stp off
1417          bridge_fd 0
1418
1419 auto vxlan2
1420 iface vxlan2 inet manual
1421         vxlan-id 2
1422         vxlan-local-tunnelip 192.168.0.3
1423         bridge-learning off
1424         bridge-arp-nd-suppress on
1425         bridge-unicast-flood off
1426         bridge-multicast-flood off
1427
1428 auto vmbr2
1429 iface vmbr2 inet static
1430         bridge_ports vxlan2
1431         bridge_stp off
1432         bridge_fd 0
1433         address 10.0.2.254
1434         netmask 255.255.255.0
1435         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1436         vrf vrf1
1437
1438 auto vxlan3
1439 iface vxlan3 inet manual
1440         vxlan-id 3
1441         vxlan-local-tunnelip 192.168.0.3
1442         bridge-learning off
1443         bridge-arp-nd-suppress on
1444         bridge-unicast-flood off
1445         bridge-multicast-flood off
1446
1447 auto vmbr3
1448 iface vmbr3 inet static
1449         bridge_ports vxlan3
1450         bridge_stp off
1451         bridge_fd 0
1452         address 10.0.3.254
1453         netmask 255.255.255.0
1454         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1455         vrf vrf1
1456
1457 #interconnect vxlan-vfr l3vni
1458 auto vxlan4000
1459 iface vxlan4000 inet manual
1460         vxlan-id 4000
1461         vxlan-local-tunnelip 192.168.0.3
1462         bridge-learning off
1463         bridge-arp-nd-suppress on
1464         bridge-unicast-flood off
1465         bridge-multicast-flood off
1466
1467
1468 auto vmbr4000
1469 iface vmbr4000 inet manual
1470         bridge_ports vxlan4000
1471         bridge_stp off
1472         bridge_fd 0
1473         hwaddress 44:39:39:FF:40:92  #must be different on each node
1474         vrf vrf1
1475 ----
1476
1477
1478 frr.conf
1479
1480 ----
1481 vrf vrf1
1482  vni 4000
1483 !
1484 router bgp 1234
1485  bgp router-id 192.168.0.3
1486  no bgp default ipv4-unicast
1487  coalesce-time 1000
1488  neighbor 192.168.0.1 remote-as 1234
1489  neighbor 192.168.0.2 remote-as 1234
1490  !
1491  address-family l2vpn evpn
1492   neighbor 192.168.0.1 activate
1493   neighbor 192.168.0.2 activate
1494   advertise-all-vni
1495  exit-address-family
1496 !
1497 router bgp 1234 vrf vrf1
1498 !
1499  bgp router-id 192.168.0.3
1500  !
1501  address-family ipv4 unicast
1502   redistribute connected
1503  exit-address-family
1504  !
1505  address-family l2vpn evpn
1506   advertise ipv4 unicast
1507  exit-address-family
1508 !
1509 line vty
1510 !
1511 ----
1512
1513 multiple gateway nodes
1514 ^^^^^^^^^^^^^^^^^^^^^^
1515 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1516 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1517 and announce this default gw.
1518 The external router have ecmp routes to all proxmox nodes.(balancing).
1519 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1520 vxlan the packet to final destination.
1521
1522 *node1
1523
1524 ----
1525 auto vrf1
1526 iface vrf1
1527     vrf-table auto
1528
1529 auto eno1
1530 iface eno1 inet manual
1531         mtu 1550
1532
1533 auto vmbr0
1534 iface vmbr0 inet static
1535          address 192.168.0.1
1536          netmask  255.255.255.0
1537          bridge_ports eno1
1538          bridge_stp off
1539          bridge_fd 0
1540
1541 auto eno2
1542 iface eno2
1543         address 172.16.0.1
1544         netmask 255.255.255.0
1545         vrf vrf1
1546         mtu 1550
1547         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1548         #if you have multiple external routers, you can use ecmp balancing
1549         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1550
1551 auto vxlan2
1552 iface vxlan2 inet manual
1553         vxlan-id 2
1554         vxlan-local-tunnelip 192.168.0.1
1555         bridge-learning off
1556         bridge-arp-nd-suppress on
1557         bridge-unicast-flood off
1558         bridge-multicast-flood off
1559
1560 auto vmbr2
1561 iface vmbr2 inet static
1562         bridge_ports vxlan2
1563         bridge_stp off
1564         bridge_fd 0
1565         address 10.0.2.254
1566         netmask 255.255.255.0
1567         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1568         vrf vrf1
1569
1570 auto vxlan3
1571 iface vxlan3 inet manual
1572         vxlan-id 3
1573         vxlan-local-tunnelip 192.168.0.1
1574         bridge-learning off
1575         bridge-arp-nd-suppress on
1576         bridge-unicast-flood off
1577         bridge-multicast-flood off
1578
1579 auto vmbr3
1580 iface vmbr3 inet static
1581         bridge_ports vxlan3
1582         bridge_stp off
1583         bridge_fd 0
1584         address 10.0.3.254
1585         netmask 255.255.255.0
1586         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1587         vrf vrf1
1588
1589 #interconnect vxlan-vfr l3vni
1590 auto vxlan4000
1591 iface vxlan4000 inet manual
1592         vxlan-id 4000
1593         vxlan-local-tunnelip 192.168.0.1
1594         bridge-learning off
1595         bridge-arp-nd-suppress on
1596         bridge-unicast-flood off
1597         bridge-multicast-flood off
1598
1599 auto vmbr4000
1600 iface vmbr4000 inet manual
1601         bridge_ports vxlan4000
1602         bridge_stp off
1603         bridge_fd 0
1604         hwaddress 44:39:39:FF:40:90  #must be different on each node
1605         vrf vrf1
1606 ----
1607
1608
1609 frr.conf
1610
1611 ----
1612 vrf vrf1
1613  vni 4000
1614 !
1615 router bgp 1234
1616  bgp router-id 192.168.0.1
1617  no bgp default ipv4-unicast
1618  coalesce-time 1000
1619  neighbor 192.168.0.2 remote-as 1234
1620  neighbor 192.168.0.3 remote-as 1234
1621  !
1622  address-family l2vpn evpn
1623   neighbor 192.168.0.2 activate
1624   neighbor 192.168.0.3 activate
1625   advertise-all-vni
1626  exit-address-family
1627 !
1628 router bgp 1234 vrf vrf1
1629 !
1630  bgp router-id 172.16.0.1
1631  !
1632  address-family ipv4 unicast
1633   redistribute connected
1634   redistribute kernel !announce your default gw to all nodes
1635  exit-address-family
1636  !
1637  address-family l2vpn evpn
1638   advertise ipv4 unicast
1639  exit-address-family
1640 !
1641 line vty
1642 !
1643 ----
1644
1645
1646 * node2
1647
1648 ----
1649 auto vrf1
1650 iface vrf1
1651     vrf-table auto
1652
1653 auto eno1
1654 iface eno1 inet manual
1655         mtu 1550
1656
1657 auto vmbr0
1658 iface vmbr0 inet static
1659          address 192.168.0.2
1660          netmask  255.255.255.0
1661          bridge_ports eno1
1662          bridge_stp off
1663          bridge_fd 0
1664
1665 auto eno2
1666 iface eno2
1667         address 172.16.0.3
1668         netmask 255.255.255.0
1669         vrf vrf1
1670         mtu 1550
1671         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1672         #if you have multiple external routers, you can use ecmp balancing
1673         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1674
1675 auto vxlan2
1676 iface vxlan2 inet manual
1677         vxlan-id 2
1678         vxlan-local-tunnelip 192.168.0.2
1679         bridge-learning off
1680         bridge-arp-nd-suppress on
1681         bridge-unicast-flood off
1682         bridge-multicast-flood off
1683
1684 auto vmbr2
1685 iface vmbr2 inet static
1686         bridge_ports vxlan2
1687         bridge_stp off
1688         bridge_fd 0
1689         address 10.0.2.254
1690         netmask 255.255.255.0
1691         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1692         vrf vrf1
1693
1694 auto vxlan3
1695 iface vxlan3 inet manual
1696         vxlan-id 3
1697         vxlan-local-tunnelip 192.168.0.2
1698         bridge-learning off
1699         bridge-arp-nd-suppress on
1700         bridge-unicast-flood off
1701         bridge-multicast-flood off
1702
1703 auto vmbr3
1704 iface vmbr3 inet static
1705         bridge_ports vxlan3
1706         bridge_stp off
1707         bridge_fd 0
1708         address 10.0.3.254
1709         netmask 255.255.255.0
1710         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1711         vrf vrf1
1712
1713 #interconnect vxlan-vfr l3vni
1714 auto vxlan4000
1715 iface vxlan4000 inet manual
1716         vxlan-id 4000
1717         vxlan-local-tunnelip 192.168.0.2
1718         bridge-learning off
1719         bridge-arp-nd-suppress on
1720         bridge-unicast-flood off
1721         bridge-multicast-flood off
1722
1723
1724 auto vmbr4000
1725 iface vmbr4000 inet manual
1726         bridge_ports vxlan4000
1727         bridge_stp off
1728         bridge_fd 0
1729         hwaddress 44:39:39:FF:40:91  #must be different on each node
1730         vrf vrf1
1731 ----
1732
1733
1734 frr.conf
1735
1736 ----
1737 vrf vrf1
1738  vni 4000
1739 !
1740 router bgp 1234
1741  bgp router-id 192.168.0.2
1742  no bgp default ipv4-unicast
1743  coalesce-time 1000
1744  neighbor 192.168.0.1 remote-as 1234
1745  neighbor 192.168.0.3 remote-as 1234
1746  !
1747  address-family l2vpn evpn
1748   neighbor 192.168.0.1 activate
1749   neighbor 192.168.0.3 activate
1750   advertise-all-vni
1751  exit-address-family
1752 !
1753 router bgp 1234 vrf vrf1
1754 !
1755  bgp router-id 172.16.0.2
1756  !
1757  address-family ipv4 unicast
1758   redistribute connected
1759   redistribute kernel !announce your default gw to all nodes
1760  exit-address-family
1761  !
1762  address-family l2vpn evpn
1763   advertise ipv4 unicast
1764  exit-address-family
1765 !
1766 line vty
1767 !
1768 ----
1769
1770
1771 * node3
1772
1773 ----
1774 auto vrf1
1775 iface vrf1
1776     vrf-table auto
1777
1778 auto eno1
1779 iface eno1 inet manual
1780         mtu 1550
1781
1782 auto vmbr0
1783 iface vmbr0 inet static
1784          address 192.168.0.3
1785          netmask  255.255.255.0
1786          bridge_ports eno1
1787          bridge_stp off
1788          bridge_fd 0
1789
1790 auto eno2
1791 iface eno2
1792         address 172.16.0.3
1793         netmask 255.255.255.0
1794         vrf vrf1
1795         mtu 1550
1796         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1797         #if you have multiple external routers, you can use ecmp balancing
1798         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1799
1800 auto vxlan2
1801 iface vxlan2 inet manual
1802         vxlan-id 2
1803         vxlan-local-tunnelip 192.168.0.3
1804         bridge-learning off
1805         bridge-arp-nd-suppress on
1806         bridge-unicast-flood off
1807         bridge-multicast-flood off
1808
1809 auto vmbr2
1810 iface vmbr2 inet static
1811         bridge_ports vxlan2
1812         bridge_stp off
1813         bridge_fd 0
1814         address 10.0.2.254
1815         netmask 255.255.255.0
1816         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1817         vrf vrf1
1818
1819 auto vxlan3
1820 iface vxlan3 inet manual
1821         vxlan-id 3
1822         vxlan-local-tunnelip 192.168.0.3
1823         bridge-learning off
1824         bridge-arp-nd-suppress on
1825         bridge-unicast-flood off
1826         bridge-multicast-flood off
1827
1828 auto vmbr3
1829 iface vmbr3 inet static
1830         bridge_ports vxlan3
1831         bridge_stp off
1832         bridge_fd 0
1833         address 10.0.3.254
1834         netmask 255.255.255.0
1835         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1836         vrf vrf1
1837
1838 #interconnect vxlan-vfr l3vni
1839 auto vxlan4000
1840 iface vxlan4000 inet manual
1841         vxlan-id 4000
1842         vxlan-local-tunnelip 192.168.0.3
1843         bridge-learning off
1844         bridge-arp-nd-suppress on
1845         bridge-unicast-flood off
1846         bridge-multicast-flood off
1847
1848
1849 auto vmbr4000
1850 iface vmbr4000 inet manual
1851         bridge_ports vxlan4000
1852         bridge_stp off
1853         bridge_fd 0
1854         hwaddress 44:39:39:FF:40:92  #must be different on each node
1855         vrf vrf1
1856 ----
1857
1858
1859 frr.conf
1860
1861 ----
1862 vrf vrf1
1863  vni 4000
1864 !
1865 router bgp 1234
1866  bgp router-id 192.168.0.3
1867  no bgp default ipv4-unicast
1868  coalesce-time 1000
1869  neighbor 192.168.0.1 remote-as 1234
1870  neighbor 192.168.0.2 remote-as 1234
1871  !
1872  address-family l2vpn evpn
1873   neighbor 192.168.0.1 activate
1874   neighbor 192.168.0.2 activate
1875   advertise-all-vni
1876  exit-address-family
1877 !
1878 router bgp 1234 vrf vrf1
1879 !
1880  bgp router-id 172.16.0.3
1881  !
1882  address-family ipv4 unicast
1883   redistribute connected
1884   redistribute kernel !announce your default gw to all nodes
1885  exit-address-family
1886  !
1887  address-family l2vpn evpn
1888   advertise ipv4 unicast
1889  exit-address-family
1890 !
1891 line vty
1892 !
1893 ----
1894
1895 Note
1896 ^^^^
1897
1898 If your external router don't support ecmp to reach multiple proxmox nodes,
1899 you can setup an HA floating vip on proxmox nodes with vrrp
1900
1901 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1902 Node1 is the primary and failover to node2 in case of failure.
1903
1904
1905 * node1
1906
1907 ----
1908 auto eno2
1909 iface eno2
1910         address 172.16.0.1
1911         netmask 255.255.255.0
1912         vrf vrf1
1913         mtu 1550
1914         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1915         vrrp-id 1
1916         vrrp-priority 1
1917         vrrp-virtual-ip 172.16.0.10
1918 ----
1919
1920 * node2
1921
1922 ----
1923 auto eno2
1924 iface eno2
1925         address 172.16.0.2
1926         netmask 255.255.255.0
1927         mtu 1550
1928         vrf vrf1
1929         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1930         vrrp-id 1
1931         vrrp-priority 2
1932         vrrp-virtual-ip 172.16.0.10
1933 ----
1934
1935