vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  coalesce-time 1000
 371  neighbor 192.168.0.2 remote-as 1234
 372  neighbor 192.168.0.3 remote-as 1234
 373  !
 374  address-family l2vpn evpn
 375   neighbor 192.168.0.2 activate
 376   neighbor 192.168.0.3 activate
 377   advertise-all-vni
 378  exit-address-family
 379 !
 380 line vty
 381 !
 382 ----
 383
 384
 385 * node2
 386
 387 ----
 388 auto eno1
 389 iface eno1 inet manual
 390         mtu 1550
 391
 392 auto vmbr0
 393 iface vmbr0 inet static
 394         address 192.168.0.2
 395         netmask 255.255.255.0
 396         bridge_ports eno1
 397         bridge_stp off
 398         bridge_fd 0
 399
 400 auto vxlan2
 401 iface vxlan2 inet manual
 402         vxlan-id 2
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr2
 411 iface vmbr2 inet manual
 412         bridge_ports vxlan2
 413         bridge_stp off
 414         bridge_fd 0
 415
 416 auto vxlan3
 417 iface vxlan3 inet manual
 418         vxlan-id 3
 419         vxlan-local-tunnelip 192.168.0.2
 420         bridge-learning off
 421         bridge-arp-nd-suppress on
 422         bridge-unicast-flood off
 423         bridge-multicast-flood off
 424
 425
 426 auto vmbr3
 427 iface vmbr3 inet manual
 428         bridge_ports vxlan3
 429         bridge_stp off
 430         bridge_fd 0
 431 ----
 432
 433
 434 /etc/frr/frr.conf
 435
 436 ----
 437 router bgp 1234
 438  no bgp default ipv4-unicast
 439  coalesce-time 1000
 440  neighbor 192.168.0.1 remote-as 1234
 441  neighbor 192.168.0.3 remote-as 1234
 442  !
 443  address-family l2vpn evpn
 444   neighbor 192.168.0.1 activate
 445   neighbor 192.168.0.3 activate
 446   advertise-all-vni
 447  exit-address-family
 448 !
 449 line vty
 450 !
 451 ----
 452
 453
 454 * node3
 455
 456 ----
 457 auto eno1
 458 iface eno1 inet manual
 459         mtu 1550
 460
 461 auto vmbr0
 462 iface vmbr0 inet static
 463         address 192.168.0.2
 464         netmask 255.255.255.0
 465         bridge_ports eno1
 466         bridge_stp off
 467         bridge_fd 0
 468
 469 auto vxlan2
 470 iface vxlan2 inet manual
 471         vxlan-id 2
 472         vxlan-local-tunnelip 192.168.0.3
 473         bridge-learning off
 474         bridge-arp-nd-suppress on
 475         bridge-unicast-flood off
 476         bridge-multicast-flood off
 477
 478
 479 auto vmbr2
 480 iface vmbr2 inet manual
 481         bridge_ports vxlan2
 482         bridge_stp off
 483         bridge_fd 0
 484
 485 auto vxlan3
 486 iface vxlan3 inet manual
 487         vxlan-id 3
 488         vxlan-local-tunnelip 192.168.0.3
 489         bridge-learning off
 490         bridge-arp-nd-suppress on
 491         bridge-unicast-flood off
 492         bridge-multicast-flood off
 493
 494
 495 auto vmbr3
 496 iface vmbr3 inet manual
 497         bridge_ports vxlan3
 498         bridge_stp off
 499         bridge_fd 0
 500 ----
 501
 502
 503 /etc/frr/frr.conf
 504
 505
 506 ----
 507 router bgp 1234
 508  no bgp default ipv4-unicast
 509  coalesce-time 1000
 510  neighbor 192.168.0.1 remote-as 1234
 511  neighbor 192.168.0.2 remote-as 1234
 512  !
 513  address-family l2vpn evpn
 514   neighbor 192.168.0.1 activate
 515   neighbor 192.168.0.2 activate
 516   advertise-all-vni
 517  exit-address-family
 518 !
 519 line vty
 520 !
 521 ----
 522
 523 VXLAN layer3 routing with anycast gateway
 524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 525
 526 With this need, each vmbr bridge will be the gateway for the vm.
 527 Same vmbr on different node, will have same ip address and same mac address,
 528 to have working vm live migration and no network disruption.
 529
 530 VXLAN layer3 routing only work with FRR and non-aware bridge.
 531 (vlan aware bridge support is buggy currently).
 532
 533 asymmetric model
 534 ^^^^^^^^^^^^^^^^
 535
 536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 537
 538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 539 but only bridging on the egress.
 540 This results in bi-directional VXLAN traffic traveling on different VNIs
 541 in each direction (always the destination VNI) across the routed infrastructure.
 542
 543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 544
 545
 546 sysctl.conf tuning
 547
 548 ----
 549 #enable routing
 550 net.ipv4.ip_forward=1
 551 net.ipv6.conf.all.forwarding=1
 552 ----
 553
 554 * node1
 555
 556 ----
 557 auto eno1
 558 iface eno1 inet manual
 559         mtu 1550
 560
 561 auto vmbr0
 562 iface vmbr0 inet static
 563          address 192.168.0.1
 564          netmask  255.255.255.0
 565          bridge_ports eno1
 566          bridge_stp off
 567          bridge_fd 0
 568
 569 auto vxlan2
 570 iface vxlan2 inet manual
 571         vxlan-id 2
 572         vxlan-local-tunnelip 192.168.0.1
 573         bridge-learning off
 574         bridge-arp-nd-suppress on
 575         bridge-unicast-flood off
 576         bridge-multicast-flood off
 577
 578
 579 auto vmbr2
 580 iface vmbr2 inet static
 581         address 10.0.2.254
 582         netmask 255.255.255.0
 583         hwaddress 44:39:39:FF:40:94
 584         bridge_ports vxlan2
 585         bridge_stp off
 586         bridge_fd 0
 587
 588
 589 auto vxlan3
 590 iface vxlan3 inet manual
 591         vxlan-id 3
 592         vxlan-local-tunnelip 192.168.0.1
 593         bridge-learning off
 594         bridge-arp-nd-suppress on
 595         bridge-unicast-flood off
 596         bridge-multicast-flood off
 597
 598
 599 auto vmbr3
 600 iface vmbr3 inet static
 601         address 10.0.3.254
 602         netmask 255.255.255.0
 603         hwaddress 44:39:39:FF:40:94
 604         bridge_ports vxlan3
 605         bridge_stp off
 606         bridge_fd 0
 607 ----
 608
 609
 610 frr.conf
 611
 612 ----
 613 router bgp 1234
 614  bgp router-id 192.168.0.1
 615  no bgp default ipv4-unicast
 616  coalesce-time 1000
 617  neighbor 192.168.0.2 remote-as 1234
 618  neighbor 192.168.0.3 remote-as 1234
 619  !
 620  address-family l2vpn evpn
 621   neighbor 192.168.0.2 activate
 622   neighbor 192.168.0.3 activate
 623   advertise-all-vni
 624  exit-address-family
 625 !
 626 line vty
 627 !
 628 ----
 629
 630
 631 * node2
 632
 633 ----
 634 auto eno1
 635 iface eno1 inet manual
 636         mtu 1550
 637
 638 auto vmbr0
 639 iface vmbr0 inet static
 640          address 192.168.0.2
 641          netmask  255.255.255.0
 642          bridge_ports eno1
 643          bridge_stp off
 644          bridge_fd 0
 645
 646 auto vxlan2
 647 iface vxlan2 inet manual
 648         vxlan-id 2
 649         vxlan-local-tunnelip 192.168.0.2
 650         bridge-learning off
 651         bridge-arp-nd-suppress on
 652         bridge-unicast-flood off
 653         bridge-multicast-flood off
 654
 655
 656 auto vmbr2
 657 iface vmbr2 inet static
 658         address 10.0.2.254
 659         netmask 255.255.255.0
 660         hwaddress 44:39:39:FF:40:94
 661         bridge_ports vxlan2
 662         bridge_stp off
 663         bridge_fd 0
 664
 665
 666 auto vxlan3
 667 iface vxlan3 inet manual
 668         vxlan-id 3
 669         vxlan-local-tunnelip 192.168.0.2
 670         bridge-learning off
 671         bridge-arp-nd-suppress on
 672         bridge-unicast-flood off
 673         bridge-multicast-flood off
 674
 675
 676 auto vmbr3
 677 iface vmbr3 inet static
 678         address 10.0.3.254
 679         netmask 255.255.255.0
 680         hwaddress 44:39:39:FF:40:94
 681         bridge_ports vxlan3
 682         bridge_stp off
 683         bridge_fd 0
 684 ----
 685
 686
 687 frr.conf
 688
 689 ----
 690 router bgp 1234
 691  bgp router-id 192.168.0.2
 692  no bgp default ipv4-unicast
 693  coalesce-time 1000
 694  neighbor 192.168.0.1 remote-as 1234
 695  neighbor 192.168.0.3 remote-as 1234
 696  !
 697  address-family l2vpn evpn
 698   neighbor 192.168.0.1 activate
 699   neighbor 192.168.0.3 activate
 700   advertise-all-vni
 701  exit-address-family
 702 !
 703 line vty
 704 !
 705 ----
 706
 707
 708 * node3
 709
 710 ----
 711 auto eno1
 712 iface eno1 inet manual
 713         mtu 1550
 714
 715 auto vmbr0
 716 iface vmbr0 inet static
 717          address 192.168.0.3
 718          netmask  255.255.255.0
 719          bridge_ports eno1
 720          bridge_stp off
 721          bridge_fd 0
 722
 723 auto vxlan2
 724 iface vxlan2 inet manual
 725         vxlan-id 2
 726         vxlan-local-tunnelip 192.168.0.3
 727         bridge-learning off
 728         bridge-arp-nd-suppress on
 729         bridge-unicast-flood off
 730         bridge-multicast-flood off
 731
 732
 733 auto vmbr2
 734 iface vmbr2 inet static
 735         address 10.0.2.254
 736         netmask 255.255.255.0
 737         hwaddress 44:39:39:FF:40:94
 738         bridge_ports vxlan2
 739         bridge_stp off
 740         bridge_fd 0
 741
 742
 743 auto vxlan3
 744 iface vxlan3 inet manual
 745         vxlan-id 3
 746         vxlan-local-tunnelip 192.168.0.3
 747         bridge-learning off
 748         bridge-arp-nd-suppress on
 749         bridge-unicast-flood off
 750         bridge-multicast-flood off
 751
 752
 753 auto vmbr3
 754 iface vmbr3 inet static
 755         address 10.0.3.254
 756         netmask 255.255.255.0
 757         hwaddress 44:39:39:FF:40:94
 758         bridge_ports vxlan3
 759         bridge_stp off
 760         bridge_fd 0
 761 ----
 762
 763
 764 frr.conf
 765
 766 ----
 767 router bgp 1234
 768  bgp router-id 192.168.0.3
 769  no bgp default ipv4-unicast
 770  coalesce-time 1000
 771  neighbor 192.168.0.1 remote-as 1234
 772  neighbor 192.168.0.2 remote-as 1234
 773  !
 774  address-family l2vpn evpn
 775   neighbor 192.168.0.1 activate
 776   neighbor 192.168.0.2 activate
 777   advertise-all-vni
 778  exit-address-family
 779 !
 780 line vty
 781 !
 782 ----
 783
 784
 785 symmetric model
 786 ^^^^^^^^^^^^^^^
 787
 788 With this model, you don't need to have all vxlan on all nodes.
 789 This model will also be needed to route traffic to an external router.
 790
 791 The symmetric model routes and bridges on both the ingress and the egress leafs.
 792 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 793 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 794 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 795 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 796
 797 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 798
 799 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 800
 801 sysctl.conf tuning
 802
 803 ----
 804 #enable routing
 805 net.ipv4.ip_forward=1
 806 net.ipv6.conf.all.forwarding=1
 807 ----
 808
 809 * node1
 810
 811 ----
 812 auto vrf1
 813 iface vrf1
 814     vrf-table auto
 815
 816 auto eno1
 817 iface eno1 inet manual
 818         mtu 1550
 819
 820 auto vmbr0
 821 iface vmbr0 inet static
 822          address 192.168.0.1
 823          netmask  255.255.255.0
 824          bridge_ports eno1
 825          bridge_stp off
 826          bridge_fd 0
 827
 828 auto vxlan2
 829 iface vxlan2 inet manual
 830         vxlan-id 2
 831         vxlan-local-tunnelip 192.168.0.1
 832         bridge-learning off
 833         bridge-arp-nd-suppress on
 834         bridge-unicast-flood off
 835         bridge-multicast-flood off
 836
 837 auto vmbr2
 838 iface vmbr2 inet static
 839         bridge_ports vxlan2
 840         bridge_stp off
 841         bridge_fd 0
 842         address 10.0.2.254
 843         netmask 255.255.255.0
 844         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 845         vrf vrf1
 846
 847 auto vxlan3
 848 iface vxlan3 inet manual
 849         vxlan-id 3
 850         vxlan-local-tunnelip 192.168.0.1
 851         bridge-learning off
 852         bridge-arp-nd-suppress on
 853         bridge-unicast-flood off
 854         bridge-multicast-flood off
 855
 856 auto vmbr3
 857 iface vmbr3 inet static
 858         bridge_ports vxlan3
 859         bridge_stp off
 860         bridge_fd 0
 861         address 10.0.3.254
 862         netmask 255.255.255.0
 863         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 864         vrf vrf1
 865
 866 #interconnect vxlan-vfr l3vni
 867 auto vxlan4000
 868 iface vxlan4000 inet manual
 869         vxlan-id 4000
 870         vxlan-local-tunnelip 192.168.0.1
 871         bridge-learning off
 872         bridge-arp-nd-suppress on
 873         bridge-unicast-flood off
 874         bridge-multicast-flood off
 875
 876
 877 auto vmbr4000
 878 iface vmbr4000 inet manual
 879         bridge_ports vxlan4000
 880         bridge_stp off
 881         bridge_fd 0
 882         vrf vrf1
 883 ----
 884
 885 frr.conf
 886
 887 ----
 888 vrf vrf1
 889  vni 4000
 890  exit-vrf
 891 !
 892 router bgp 1234
 893  bgp router-id 192.168.0.1
 894  no bgp default ipv4-unicast
 895  coalesce-time 1000
 896  neighbor 192.168.0.2 remote-as 1234
 897  neighbor 192.168.0.3 remote-as 1234
 898  !
 899  address-family l2vpn evpn
 900   neighbor 192.168.0.2 activate
 901   neighbor 192.168.0.3 activate
 902   advertise-all-vni
 903  exit-address-family
 904 !
 905 line vty
 906 !
 907 ----
 908
 909
 910 * node2
 911
 912 ----
 913 auto vrf1
 914 iface vrf1
 915     vrf-table auto
 916
 917 auto eno1
 918 iface eno1 inet manual
 919         mtu 1550
 920
 921 auto vmbr0
 922 iface vmbr0 inet static
 923          address 192.168.0.2
 924          netmask  255.255.255.0
 925          bridge_ports eno1
 926          bridge_stp off
 927          bridge_fd 0
 928
 929 auto vxlan2
 930 iface vxlan2 inet manual
 931         vxlan-id 2
 932         vxlan-local-tunnelip 192.168.0.2
 933         bridge-learning off
 934         bridge-arp-nd-suppress on
 935         bridge-unicast-flood off
 936         bridge-multicast-flood off
 937
 938 auto vmbr2
 939 iface vmbr2 inet static
 940         bridge_ports vxlan2
 941         bridge_stp off
 942         bridge_fd 0
 943         address 10.0.2.254
 944         netmask 255.255.255.0
 945         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 946         vrf vrf1
 947
 948 auto vxlan3
 949 iface vxlan3 inet manual
 950         vxlan-id 3
 951         vxlan-local-tunnelip 192.168.0.2
 952         bridge-learning off
 953         bridge-arp-nd-suppress on
 954         bridge-unicast-flood off
 955         bridge-multicast-flood off
 956
 957 auto vmbr3
 958 iface vmbr3 inet static
 959         bridge_ports vxlan3
 960         bridge_stp off
 961         bridge_fd 0
 962         address 10.0.3.254
 963         netmask 255.255.255.0
 964         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 965         vrf vrf1
 966
 967 #interconnect vxlan-vfr l3vni
 968 auto vxlan4000
 969 iface vxlan4000 inet manual
 970         vxlan-id 4000
 971         vxlan-local-tunnelip 192.168.0.2
 972         bridge-learning off
 973         bridge-arp-nd-suppress on
 974         bridge-unicast-flood off
 975         bridge-multicast-flood off
 976
 977
 978 auto vmbr4000
 979 iface vmbr4000 inet manual
 980         bridge_ports vxlan4000
 981         bridge_stp off
 982         bridge_fd 0
 983         vrf vrf1
 984 ----
 985
 986
 987 frr.conf
 988
 989 ----
 990 vrf vrf1
 991  vni 4000
 992  exit-vrf
 993 !
 994 router bgp 1234
 995  bgp router-id 192.168.0.2
 996  no bgp default ipv4-unicast
 997  coalesce-time 1000
 998  neighbor 192.168.0.1 remote-as 1234
 999  neighbor 192.168.0.3 remote-as 1234
1000  !
1001  address-family l2vpn evpn
1002   neighbor 192.168.0.1 activate
1003   neighbor 192.168.0.3 activate
1004   advertise-all-vni
1005  exit-address-family
1006 !
1007 line vty
1008 !
1009 ----
1010
1011
1012 * node3
1013
1014 ----
1015 auto vrf1
1016 iface vrf1
1017     vrf-table auto
1018
1019 auto eno1
1020 iface eno1 inet manual
1021         mtu 1550
1022
1023 auto vmbr0
1024 iface vmbr0 inet static
1025          address 192.168.0.3
1026          netmask  255.255.255.0
1027          bridge_ports eno1
1028          bridge_stp off
1029          bridge_fd 0
1030
1031 auto vxlan2
1032 iface vxlan2 inet manual
1033         vxlan-id 2
1034         vxlan-local-tunnelip 192.168.0.3
1035         bridge-learning off
1036         bridge-arp-nd-suppress on
1037         bridge-unicast-flood off
1038         bridge-multicast-flood off
1039
1040 auto vmbr2
1041 iface vmbr2 inet static
1042         bridge_ports vxlan2
1043         bridge_stp off
1044         bridge_fd 0
1045         address 10.0.2.254
1046         netmask 255.255.255.0
1047         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1048         vrf vrf1
1049
1050 auto vxlan3
1051 iface vxlan3 inet manual
1052         vxlan-id 3
1053         vxlan-local-tunnelip 192.168.0.3
1054         bridge-learning off
1055         bridge-arp-nd-suppress on
1056         bridge-unicast-flood off
1057         bridge-multicast-flood off
1058
1059 auto vmbr3
1060 iface vmbr3 inet static
1061         bridge_ports vxlan3
1062         bridge_stp off
1063         bridge_fd 0
1064         address 10.0.3.254
1065         netmask 255.255.255.0
1066         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1067         vrf vrf1
1068
1069 #interconnect vxlan-vfr l3vni
1070 auto vxlan4000
1071 iface vxlan4000 inet manual
1072         vxlan-id 4000
1073         vxlan-local-tunnelip 192.168.0.3
1074         bridge-learning off
1075         bridge-arp-nd-suppress on
1076         bridge-unicast-flood off
1077         bridge-multicast-flood off
1078
1079
1080 auto vmbr4000
1081 iface vmbr4000 inet manual
1082         bridge_ports vxlan4000
1083         bridge_stp off
1084         bridge_fd 0
1085         vrf vrf1
1086 ----
1087
1088
1089 frr.conf
1090
1091 ----
1092 vrf vrf1
1093  vni 4000
1094  exit-vrf
1095 !
1096 router bgp 1234
1097  bgp router-id 192.168.0.3
1098  no bgp default ipv4-unicast
1099  coalesce-time 1000
1100  neighbor 192.168.0.1 remote-as 1234
1101  neighbor 192.168.0.2 remote-as 1234
1102  !
1103  address-family l2vpn evpn
1104   neighbor 192.168.0.1 activate
1105   neighbor 192.168.0.2 activate
1106   advertise-all-vni
1107  exit-address-family
1108 !
1109 line vty
1110 !
1111 ----
1112
1113 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1114 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1115 Routing to outside need the symmetric model.
1116
1117 1 gateway node
1118 ^^^^^^^^^^^^^^
1119 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1120 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1121
1122
1123 *node1
1124
1125 ----
1126 auto vrf1
1127 iface vrf1
1128     vrf-table auto
1129
1130 auto eno1
1131 iface eno1 inet manual
1132         mtu 1550
1133
1134 auto vmbr0
1135 iface vmbr0 inet static
1136          address 192.168.0.1
1137          netmask  255.255.255.0
1138          gateway 192.168.0.254
1139          bridge_ports eno1
1140          bridge_stp off
1141          bridge_fd 0
1142
1143 auto vxlan2
1144 iface vxlan2 inet manual
1145         vxlan-id 2
1146         vxlan-local-tunnelip 192.168.0.1
1147         bridge-learning off
1148         bridge-arp-nd-suppress on
1149         bridge-unicast-flood off
1150         bridge-multicast-flood off
1151
1152 auto vmbr2
1153 iface vmbr2 inet static
1154         bridge_ports vxlan2
1155         bridge_stp off
1156         bridge_fd 0
1157         address 10.0.2.254
1158         netmask 255.255.255.0
1159         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1160         vrf vrf1
1161
1162 auto vxlan3
1163 iface vxlan3 inet manual
1164         vxlan-id 3
1165         vxlan-local-tunnelip 192.168.0.1
1166         bridge-learning off
1167         bridge-arp-nd-suppress on
1168         bridge-unicast-flood off
1169         bridge-multicast-flood off
1170
1171 auto vmbr3
1172 iface vmbr3 inet static
1173         bridge_ports vxlan3
1174         bridge_stp off
1175         bridge_fd 0
1176         address 10.0.3.254
1177         netmask 255.255.255.0
1178         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1179         vrf vrf1
1180
1181 #interconnect vxlan-vfr l3vni
1182 auto vxlan4000
1183 iface vxlan4000 inet manual
1184         vxlan-id 4000
1185         vxlan-local-tunnelip 192.168.0.1
1186         bridge-learning off
1187         bridge-arp-nd-suppress on
1188         bridge-unicast-flood off
1189         bridge-multicast-flood off
1190
1191 auto vmbr4000
1192 iface vmbr4000 inet manual
1193         bridge_ports vxlan4000
1194         bridge_stp off
1195         bridge_fd 0
1196         vrf vrf1
1197 ----
1198
1199
1200 frr.conf
1201
1202 ----
1203 vrf vrf1
1204  vni 4000
1205  exit-vrf
1206 !
1207 router bgp 1234
1208  bgp router-id 192.168.0.1
1209  no bgp default ipv4-unicast
1210  coalesce-time 1000
1211  neighbor 192.168.0.2 remote-as 1234
1212  neighbor 192.168.0.3 remote-as 1234
1213  !
1214  address-family ipv4 unicast
1215   import vrf vrf1
1216  exit-address-family
1217  !
1218  address-family l2vpn evpn
1219   neighbor 192.168.0.2 activate
1220   neighbor 192.168.0.3 activate
1221   advertise-all-vni
1222  exit-address-family
1223 !
1224 router bgp 1234 vrf vrf1
1225 !
1226  address-family l2vpn evpn
1227   default-originate ipv4
1228  exit-address-family
1229 !
1230 line vty
1231 !
1232 ----
1233
1234
1235 * node2
1236
1237 ----
1238 auto vrf1
1239 iface vrf1
1240     vrf-table auto
1241
1242 auto eno1
1243 iface eno1 inet manual
1244         mtu 1550
1245
1246 auto vmbr0
1247 iface vmbr0 inet static
1248          address 192.168.0.2
1249          netmask  255.255.255.0
1250          bridge_ports eno1
1251          bridge_stp off
1252          bridge_fd 0
1253
1254 auto vxlan2
1255 iface vxlan2 inet manual
1256         vxlan-id 2
1257         vxlan-local-tunnelip 192.168.0.2
1258         bridge-learning off
1259         bridge-arp-nd-suppress on
1260         bridge-unicast-flood off
1261         bridge-multicast-flood off
1262
1263 auto vmbr2
1264 iface vmbr2 inet static
1265         bridge_ports vxlan2
1266         bridge_stp off
1267         bridge_fd 0
1268         address 10.0.2.254
1269         netmask 255.255.255.0
1270         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1271         vrf vrf1
1272
1273 auto vxlan3
1274 iface vxlan3 inet manual
1275         vxlan-id 3
1276         vxlan-local-tunnelip 192.168.0.2
1277         bridge-learning off
1278         bridge-arp-nd-suppress on
1279         bridge-unicast-flood off
1280         bridge-multicast-flood off
1281
1282 auto vmbr3
1283 iface vmbr3 inet static
1284         bridge_ports vxlan3
1285         bridge_stp off
1286         bridge_fd 0
1287         address 10.0.3.254
1288         netmask 255.255.255.0
1289         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1290         vrf vrf1
1291
1292 #interconnect vxlan-vfr l3vni
1293 auto vxlan4000
1294 iface vxlan4000 inet manual
1295         vxlan-id 4000
1296         vxlan-local-tunnelip 192.168.0.2
1297         bridge-learning off
1298         bridge-arp-nd-suppress on
1299         bridge-unicast-flood off
1300         bridge-multicast-flood off
1301
1302
1303 auto vmbr4000
1304 iface vmbr4000 inet manual
1305         bridge_ports vxlan4000
1306         bridge_stp off
1307         bridge_fd 0
1308         vrf vrf1
1309 ----
1310
1311
1312 frr.conf
1313
1314 ----
1315 vrf vrf1
1316  vni 4000
1317  exit-vrf
1318 !
1319 router bgp 1234
1320  bgp router-id 192.168.0.2
1321  no bgp default ipv4-unicast
1322  coalesce-time 1000
1323  neighbor 192.168.0.1 remote-as 1234
1324  neighbor 192.168.0.3 remote-as 1234
1325  !
1326  address-family l2vpn evpn
1327   neighbor 192.168.0.1 activate
1328   neighbor 192.168.0.3 activate
1329   advertise-all-vni
1330  exit-address-family
1331 !
1332 line vty
1333 !
1334 ----
1335
1336
1337 * node3
1338
1339 ----
1340 auto vrf1
1341 iface vrf1
1342     vrf-table auto
1343
1344 auto eno1
1345 iface eno1 inet manual
1346         mtu 1550
1347
1348 auto vmbr0
1349 iface vmbr0 inet static
1350          address 192.168.0.3
1351          netmask  255.255.255.0
1352          bridge_ports eno1
1353          bridge_stp off
1354          bridge_fd 0
1355
1356 auto vxlan2
1357 iface vxlan2 inet manual
1358         vxlan-id 2
1359         vxlan-local-tunnelip 192.168.0.3
1360         bridge-learning off
1361         bridge-arp-nd-suppress on
1362         bridge-unicast-flood off
1363         bridge-multicast-flood off
1364
1365 auto vmbr2
1366 iface vmbr2 inet static
1367         bridge_ports vxlan2
1368         bridge_stp off
1369         bridge_fd 0
1370         address 10.0.2.254
1371         netmask 255.255.255.0
1372         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1373         vrf vrf1
1374
1375 auto vxlan3
1376 iface vxlan3 inet manual
1377         vxlan-id 3
1378         vxlan-local-tunnelip 192.168.0.3
1379         bridge-learning off
1380         bridge-arp-nd-suppress on
1381         bridge-unicast-flood off
1382         bridge-multicast-flood off
1383
1384 auto vmbr3
1385 iface vmbr3 inet static
1386         bridge_ports vxlan3
1387         bridge_stp off
1388         bridge_fd 0
1389         address 10.0.3.254
1390         netmask 255.255.255.0
1391         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1392         vrf vrf1
1393
1394 #interconnect vxlan-vfr l3vni
1395 auto vxlan4000
1396 iface vxlan4000 inet manual
1397         vxlan-id 4000
1398         vxlan-local-tunnelip 192.168.0.3
1399         bridge-learning off
1400         bridge-arp-nd-suppress on
1401         bridge-unicast-flood off
1402         bridge-multicast-flood off
1403
1404
1405 auto vmbr4000
1406 iface vmbr4000 inet manual
1407         bridge_ports vxlan4000
1408         bridge_stp off
1409         bridge_fd 0
1410         vrf vrf1
1411 ----
1412
1413
1414 frr.conf
1415
1416 ----
1417 vrf vrf1
1418  vni 4000
1419  exit-vrf
1420 !
1421 router bgp 1234
1422  bgp router-id 192.168.0.3
1423  no bgp default ipv4-unicast
1424  coalesce-time 1000
1425  neighbor 192.168.0.1 remote-as 1234
1426  neighbor 192.168.0.2 remote-as 1234
1427  !
1428  address-family l2vpn evpn
1429   neighbor 192.168.0.1 activate
1430   neighbor 192.168.0.2 activate
1431   advertise-all-vni
1432  exit-address-family
1433 !
1434 line vty
1435 !
1436 ----
1437
1438 multiple gateway nodes
1439 ^^^^^^^^^^^^^^^^^^^^^^
1440 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1441 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1442 and announce this default gw in the vrf (default originate)
1443 The external router have ecmp routes to all proxmox nodes.(balancing).
1444 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1445 vxlan the packet to final destination.
1446
1447 *node1
1448
1449 ----
1450 auto vrf1
1451 iface vrf1
1452     vrf-table auto
1453
1454 auto eno1
1455 iface eno1 inet manual
1456         mtu 1550
1457
1458 auto vmbr0
1459 iface vmbr0 inet static
1460          address 192.168.0.1
1461          netmask  255.255.255.0
1462          gateway 192.168.0.254
1463          bridge_ports eno1
1464          bridge_stp off
1465          bridge_fd 0
1466
1467 auto vxlan2
1468 iface vxlan2 inet manual
1469         vxlan-id 2
1470         vxlan-local-tunnelip 192.168.0.1
1471         bridge-learning off
1472         bridge-arp-nd-suppress on
1473         bridge-unicast-flood off
1474         bridge-multicast-flood off
1475
1476 auto vmbr2
1477 iface vmbr2 inet static
1478         bridge_ports vxlan2
1479         bridge_stp off
1480         bridge_fd 0
1481         address 10.0.2.254
1482         netmask 255.255.255.0
1483         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1484         vrf vrf1
1485
1486 auto vxlan3
1487 iface vxlan3 inet manual
1488         vxlan-id 3
1489         vxlan-local-tunnelip 192.168.0.1
1490         bridge-learning off
1491         bridge-arp-nd-suppress on
1492         bridge-unicast-flood off
1493         bridge-multicast-flood off
1494
1495 auto vmbr3
1496 iface vmbr3 inet static
1497         bridge_ports vxlan3
1498         bridge_stp off
1499         bridge_fd 0
1500         address 10.0.3.254
1501         netmask 255.255.255.0
1502         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1503         vrf vrf1
1504
1505 #interconnect vxlan-vfr l3vni
1506 auto vxlan4000
1507 iface vxlan4000 inet manual
1508         vxlan-id 4000
1509         vxlan-local-tunnelip 192.168.0.1
1510         bridge-learning off
1511         bridge-arp-nd-suppress on
1512         bridge-unicast-flood off
1513         bridge-multicast-flood off
1514
1515 auto vmbr4000
1516 iface vmbr4000 inet manual
1517         bridge_ports vxlan4000
1518         bridge_stp off
1519         bridge_fd 0
1520         vrf vrf1
1521 ----
1522
1523
1524 frr.conf
1525
1526 ----
1527 vrf vrf1
1528  vni 4000
1529  exit-vrf
1530 !
1531 router bgp 1234
1532  bgp router-id 192.168.0.1
1533  no bgp default ipv4-unicast
1534  coalesce-time 1000
1535  neighbor 192.168.0.2 remote-as 1234
1536  neighbor 192.168.0.3 remote-as 1234
1537  !
1538  address-family ipv4 unicast
1539   import vrf vrf1
1540  exit-address-family
1541  !
1542  address-family l2vpn evpn
1543   neighbor 192.168.0.2 activate
1544   neighbor 192.168.0.3 activate
1545   advertise-all-vni
1546  exit-address-family
1547 !
1548 router bgp 1234 vrf vrf1
1549 !
1550  address-family l2vpn evpn
1551   default-originate ipv4
1552  exit-address-family
1553 !
1554 line vty
1555 !
1556 ----
1557
1558
1559 * node2
1560
1561 ----
1562 auto vrf1
1563 iface vrf1
1564     vrf-table auto
1565
1566 auto eno1
1567 iface eno1 inet manual
1568         mtu 1550
1569
1570 auto vmbr0
1571 iface vmbr0 inet static
1572          address 192.168.0.2
1573          netmask  255.255.255.0
1574          gateway 192.168.0.254
1575          bridge_ports eno1
1576          bridge_stp off
1577          bridge_fd 0
1578
1579 auto vxlan2
1580 iface vxlan2 inet manual
1581         vxlan-id 2
1582         vxlan-local-tunnelip 192.168.0.2
1583         bridge-learning off
1584         bridge-arp-nd-suppress on
1585         bridge-unicast-flood off
1586         bridge-multicast-flood off
1587
1588 auto vmbr2
1589 iface vmbr2 inet static
1590         bridge_ports vxlan2
1591         bridge_stp off
1592         bridge_fd 0
1593         address 10.0.2.254
1594         netmask 255.255.255.0
1595         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1596         vrf vrf1
1597
1598 auto vxlan3
1599 iface vxlan3 inet manual
1600         vxlan-id 3
1601         vxlan-local-tunnelip 192.168.0.2
1602         bridge-learning off
1603         bridge-arp-nd-suppress on
1604         bridge-unicast-flood off
1605         bridge-multicast-flood off
1606
1607 auto vmbr3
1608 iface vmbr3 inet static
1609         bridge_ports vxlan3
1610         bridge_stp off
1611         bridge_fd 0
1612         address 10.0.3.254
1613         netmask 255.255.255.0
1614         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1615         vrf vrf1
1616
1617 #interconnect vxlan-vfr l3vni
1618 auto vxlan4000
1619 iface vxlan4000 inet manual
1620         vxlan-id 4000
1621         vxlan-local-tunnelip 192.168.0.2
1622         bridge-learning off
1623         bridge-arp-nd-suppress on
1624         bridge-unicast-flood off
1625         bridge-multicast-flood off
1626
1627
1628 auto vmbr4000
1629 iface vmbr4000 inet manual
1630         bridge_ports vxlan4000
1631         bridge_stp off
1632         bridge_fd 0
1633         vrf vrf1
1634 ----
1635
1636
1637 frr.conf
1638
1639 ----
1640 vrf vrf1
1641  vni 4000
1642  exit-vrf
1643 !
1644 router bgp 1234
1645  bgp router-id 192.168.0.2
1646  no bgp default ipv4-unicast
1647  coalesce-time 1000
1648  neighbor 192.168.0.1 remote-as 1234
1649  neighbor 192.168.0.3 remote-as 1234
1650  !
1651  address-family ipv4 unicast
1652   import vrf vrf1
1653  exit-address-family
1654  !
1655  address-family l2vpn evpn
1656   neighbor 192.168.0.1 activate
1657   neighbor 192.168.0.3 activate
1658   advertise-all-vni
1659  exit-address-family
1660 !
1661  address-family l2vpn evpn
1662   default-originate ipv4
1663  exit-address-family
1664 !
1665 line vty
1666 !
1667 ----
1668
1669
1670 * node3
1671
1672 ----
1673 auto vrf1
1674 iface vrf1
1675     vrf-table auto
1676
1677 auto eno1
1678 iface eno1 inet manual
1679         mtu 1550
1680
1681 auto vmbr0
1682 iface vmbr0 inet static
1683          address 192.168.0.3
1684          netmask  255.255.255.0
1685          gateway 192.168.0.254
1686          bridge_ports eno1
1687          bridge_stp off
1688          bridge_fd 0
1689
1690 auto vxlan2
1691 iface vxlan2 inet manual
1692         vxlan-id 2
1693         vxlan-local-tunnelip 192.168.0.3
1694         bridge-learning off
1695         bridge-arp-nd-suppress on
1696         bridge-unicast-flood off
1697         bridge-multicast-flood off
1698
1699 auto vmbr2
1700 iface vmbr2 inet static
1701         bridge_ports vxlan2
1702         bridge_stp off
1703         bridge_fd 0
1704         address 10.0.2.254
1705         netmask 255.255.255.0
1706         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1707         vrf vrf1
1708
1709 auto vxlan3
1710 iface vxlan3 inet manual
1711         vxlan-id 3
1712         vxlan-local-tunnelip 192.168.0.3
1713         bridge-learning off
1714         bridge-arp-nd-suppress on
1715         bridge-unicast-flood off
1716         bridge-multicast-flood off
1717
1718 auto vmbr3
1719 iface vmbr3 inet static
1720         bridge_ports vxlan3
1721         bridge_stp off
1722         bridge_fd 0
1723         address 10.0.3.254
1724         netmask 255.255.255.0
1725         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1726         vrf vrf1
1727
1728 #interconnect vxlan-vfr l3vni
1729 auto vxlan4000
1730 iface vxlan4000 inet manual
1731         vxlan-id 4000
1732         vxlan-local-tunnelip 192.168.0.3
1733         bridge-learning off
1734         bridge-arp-nd-suppress on
1735         bridge-unicast-flood off
1736         bridge-multicast-flood off
1737
1738
1739 auto vmbr4000
1740 iface vmbr4000 inet manual
1741         bridge_ports vxlan4000
1742         bridge_stp off
1743         bridge_fd 0
1744         vrf vrf1
1745 ----
1746
1747
1748 frr.conf
1749
1750 ----
1751 vrf vrf1
1752  vni 4000
1753  exit-vrf
1754 !
1755 router bgp 1234
1756  bgp router-id 192.168.0.3
1757  no bgp default ipv4-unicast
1758  coalesce-time 1000
1759  neighbor 192.168.0.1 remote-as 1234
1760  neighbor 192.168.0.2 remote-as 1234
1761  !
1762  address-family ipv4 unicast
1763   import vrf vrf1
1764  exit-address-family
1765  !
1766  address-family l2vpn evpn
1767   neighbor 192.168.0.1 activate
1768   neighbor 192.168.0.2 activate
1769   advertise-all-vni
1770  exit-address-family
1771 !
1772 router bgp 1234 vrf vrf1
1773 !
1774  address-family l2vpn evpn
1775   default-originate ipv4
1776  exit-address-family
1777 !
1778 line vty
1779 !
1780 ----
1781
1782 Note
1783 ^^^^
1784
1785 If your external router don't support ecmp static route to reach multiple proxmox nodes,
1786 you can setup an HA floating vip on proxmox nodes with vrrp
1787
1788 In this example, we will setup an floating 192.168.0.10 ip on node1 and node2.
1789 Node1 is the primary and failover to node2 in case of failure.
1790
1791 This setup need vrrpd package (apt install vrrpd).
1792 #TODO : It should be possible to do it with frr directly with last version.
1793
1794 * node1
1795
1796 ----
1797 auto vmbr0
1798 iface vmbr0 inet static
1799          address 192.168.0.1
1800          netmask  255.255.255.0
1801          gateway 192.168.0.254
1802          bridge_ports eno1
1803          bridge_stp off
1804          bridge_fd 0
1805          vrrp-id 1
1806          vrrp-priority 1
1807          vrrp-virtual-ip 192.168.0.10
1808 ----
1809
1810 * node2
1811
1812 ----
1813 auto vmbr0
1814 iface vmbr0 inet static
1815          address 192.168.0.2
1816          netmask  255.255.255.0
1817          gateway 192.168.0.254
1818          bridge_ports eno1
1819          bridge_stp off
1820          bridge_fd 0
1821          vrrp-id 1
1822          vrrp-priority 2
1823          vrrp-virtual-ip 192.168.0.10
1824 ----
1825
1826
1827 route reflectors
1828 ^^^^^^^^^^^^^^^^
1829 If you have a lot of proxmox nodes, or multiple proxmox clusters,
1830 maybe do you want to avoid that each node peer with each others nodes.
1831 For this, you can create dedicated route reflectors servers. (Minimum 2 servers for redundancy).
1832 Here an example of configuration with frr, with rrserver1 (192.168.0.200) and rrserver2 (192.168.0.201).
1833
1834
1835 rrserver1
1836 ----
1837 router bgp 1234
1838   bgp router-id 192.168.0.200
1839   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
1840   bgp log-neighbor-changes
1841   no bgp default ipv4-unicast
1842   neighbor fabric peer-group
1843   neighbor fabric remote-as 1234
1844   neighbor fabric capability extended-nexthop
1845   neighbor fabric update-source 192.168.0.200
1846   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
1847   !
1848   address-family l2vpn evpn
1849    neighbor fabric activate
1850    neighbor fabric route-reflector-client
1851    neighbor fabric allowas-in
1852   exit-address-family
1853   !
1854   exit
1855 !
1856 ---
1857
1858 rrserver2
1859 ----
1860 router bgp 1234
1861   bgp router-id 192.168.0.201
1862   bgp cluster-id 1.1.1.1
1863   bgp log-neighbor-changes
1864   no bgp default ipv4-unicast
1865   neighbor fabric peer-group
1866   neighbor fabric remote-as 1234
1867   neighbor fabric capability extended-nexthop
1868   neighbor fabric update-source 192.168.0.201
1869   bgp listen range 192.168.0.0/24 peer-group fabric
1870   !
1871   address-family l2vpn evpn
1872    neighbor fabric activate
1873    neighbor fabric route-reflector-client
1874    neighbor fabric allowas-in
1875   exit-address-family
1876   !
1877   exit
1878 !
1879 ---
1880
1881 proxmoxnode(s)
1882 ----
1883 router bgp 1234
1884  bgp router-id 192.168.0.x
1885  no bgp default ipv4-unicast
1886  coalesce-time 1000
1887  neighbor 192.168.0.200 remote-as 1234
1888  neighbor 192.168.0.201 remote-as 1234
1889  !
1890  address-family ipv4 unicast
1891   import vrf vrf1
1892  exit-address-family
1893  !
1894  address-family l2vpn evpn
1895   neighbor 192.168.0.200 activate
1896   neighbor 192.168.0.201 activate
1897   advertise-all-vni
1898  exit-address-family
1899 !
1900 ----
1901
1902 #TODO : Documentation with bgp upstream router.