vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  coalesce-time 1000
 371  neighbor 192.168.0.2 remote-as 1234
 372  neighbor 192.168.0.3 remote-as 1234
 373  !
 374  address-family l2vpn evpn
 375   neighbor 192.168.0.2 activate
 376   neighbor 192.168.0.3 activate
 377   advertise-all-vni
 378  exit-address-family
 379 !
 380 line vty
 381 !
 382 ----
 383
 384
 385 * node2
 386
 387 ----
 388 auto eno1
 389 iface eno1 inet manual
 390         mtu 1550
 391
 392 auto vmbr0
 393 iface vmbr0 inet static
 394         address 192.168.0.2
 395         netmask 255.255.255.0
 396         bridge_ports eno1
 397         bridge_stp off
 398         bridge_fd 0
 399
 400 auto vxlan2
 401 iface vxlan2 inet manual
 402         vxlan-id 2
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr2
 411 iface vmbr2 inet manual
 412         bridge_ports vxlan2
 413         bridge_stp off
 414         bridge_fd 0
 415
 416 auto vxlan3
 417 iface vxlan3 inet manual
 418         vxlan-id 3
 419         vxlan-local-tunnelip 192.168.0.2
 420         bridge-learning off
 421         bridge-arp-nd-suppress on
 422         bridge-unicast-flood off
 423         bridge-multicast-flood off
 424
 425
 426 auto vmbr3
 427 iface vmbr3 inet manual
 428         bridge_ports vxlan3
 429         bridge_stp off
 430         bridge_fd 0
 431 ----
 432
 433
 434 /etc/frr/frr.conf
 435
 436 ----
 437 router bgp 1234
 438  no bgp default ipv4-unicast
 439  coalesce-time 1000
 440  neighbor 192.168.0.1 remote-as 1234
 441  neighbor 192.168.0.3 remote-as 1234
 442  !
 443  address-family l2vpn evpn
 444   neighbor 192.168.0.1 activate
 445   neighbor 192.168.0.3 activate
 446   advertise-all-vni
 447  exit-address-family
 448 !
 449 line vty
 450 !
 451 ----
 452
 453
 454 * node3
 455
 456 ----
 457 auto eno1
 458 iface eno1 inet manual
 459         mtu 1550
 460
 461 auto vmbr0
 462 iface vmbr0 inet static
 463         address 192.168.0.2
 464         netmask 255.255.255.0
 465         bridge_ports eno1
 466         bridge_stp off
 467         bridge_fd 0
 468
 469 auto vxlan2
 470 iface vxlan2 inet manual
 471         vxlan-id 2
 472         vxlan-local-tunnelip 192.168.0.3
 473         bridge-learning off
 474         bridge-arp-nd-suppress on
 475         bridge-unicast-flood off
 476         bridge-multicast-flood off
 477
 478
 479 auto vmbr2
 480 iface vmbr2 inet manual
 481         bridge_ports vxlan2
 482         bridge_stp off
 483         bridge_fd 0
 484
 485 auto vxlan3
 486 iface vxlan3 inet manual
 487         vxlan-id 3
 488         vxlan-local-tunnelip 192.168.0.3
 489         bridge-learning off
 490         bridge-arp-nd-suppress on
 491         bridge-unicast-flood off
 492         bridge-multicast-flood off
 493
 494
 495 auto vmbr3
 496 iface vmbr3 inet manual
 497         bridge_ports vxlan3
 498         bridge_stp off
 499         bridge_fd 0
 500 ----
 501
 502
 503 /etc/frr/frr.conf
 504
 505
 506 ----
 507 router bgp 1234
 508  no bgp default ipv4-unicast
 509  coalesce-time 1000
 510  neighbor 192.168.0.1 remote-as 1234
 511  neighbor 192.168.0.2 remote-as 1234
 512  !
 513  address-family l2vpn evpn
 514   neighbor 192.168.0.1 activate
 515   neighbor 192.168.0.2 activate
 516   advertise-all-vni
 517  exit-address-family
 518 !
 519 line vty
 520 !
 521 ----
 522
 523 VXLAN layer3 routing with anycast gateway
 524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 525
 526 With this need, each vmbr bridge will be the gateway for the vm.
 527 Same vmbr on different node, will have same ip address and same mac address,
 528 to have working vm live migration and no network disruption.
 529
 530 VXLAN layer3 routing only work with FRR and non-aware bridge.
 531 (vlan aware bridge support is buggy currently).
 532
 533 asymmetric model
 534 ^^^^^^^^^^^^^^^^
 535
 536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 537
 538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 539 but only bridging on the egress.
 540 This results in bi-directional VXLAN traffic traveling on different VNIs
 541 in each direction (always the destination VNI) across the routed infrastructure.
 542
 543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 544
 545 * node1
 546
 547 ----
 548 auto eno1
 549 iface eno1 inet manual
 550         mtu 1550
 551
 552 auto vmbr0
 553 iface vmbr0 inet static
 554          address 192.168.0.1
 555          netmask  255.255.255.0
 556          bridge_ports eno1
 557          bridge_stp off
 558          bridge_fd 0
 559
 560 auto vxlan2
 561 iface vxlan2 inet manual
 562         vxlan-id 2
 563         vxlan-local-tunnelip 192.168.0.1
 564         bridge-learning off
 565         bridge-arp-nd-suppress on
 566         bridge-unicast-flood off
 567         bridge-multicast-flood off
 568
 569
 570 auto vmbr2
 571 iface vmbr2 inet static
 572         address 10.0.2.254
 573         netmask 255.255.255.0
 574         hwaddress 44:39:39:FF:40:94
 575         bridge_ports vxlan2
 576         bridge_stp off
 577         bridge_fd 0
 578         ip-forward on
 579         ip6-forward on
 580         arp-accept on
 581
 582 auto vxlan3
 583 iface vxlan3 inet manual
 584         vxlan-id 3
 585         vxlan-local-tunnelip 192.168.0.1
 586         bridge-learning off
 587         bridge-arp-nd-suppress on
 588         bridge-unicast-flood off
 589         bridge-multicast-flood off
 590
 591
 592 auto vmbr3
 593 iface vmbr3 inet static
 594         address 10.0.3.254
 595         netmask 255.255.255.0
 596         hwaddress 44:39:39:FF:40:94
 597         bridge_ports vxlan3
 598         bridge_stp off
 599         bridge_fd 0
 600         ip-forward on
 601         ip6-forward on
 602         arp-accept on
 603 ----
 604
 605
 606 frr.conf
 607
 608 ----
 609 router bgp 1234
 610  bgp router-id 192.168.0.1
 611  no bgp default ipv4-unicast
 612  coalesce-time 1000
 613  neighbor 192.168.0.2 remote-as 1234
 614  neighbor 192.168.0.3 remote-as 1234
 615  !
 616  address-family l2vpn evpn
 617   neighbor 192.168.0.2 activate
 618   neighbor 192.168.0.3 activate
 619   advertise-all-vni
 620  exit-address-family
 621 !
 622 line vty
 623 !
 624 ----
 625
 626
 627 * node2
 628
 629 ----
 630 auto eno1
 631 iface eno1 inet manual
 632         mtu 1550
 633
 634 auto vmbr0
 635 iface vmbr0 inet static
 636          address 192.168.0.2
 637          netmask  255.255.255.0
 638          bridge_ports eno1
 639          bridge_stp off
 640          bridge_fd 0
 641
 642 auto vxlan2
 643 iface vxlan2 inet manual
 644         vxlan-id 2
 645         vxlan-local-tunnelip 192.168.0.2
 646         bridge-learning off
 647         bridge-arp-nd-suppress on
 648         bridge-unicast-flood off
 649         bridge-multicast-flood off
 650
 651
 652 auto vmbr2
 653 iface vmbr2 inet static
 654         address 10.0.2.254
 655         netmask 255.255.255.0
 656         hwaddress 44:39:39:FF:40:94
 657         bridge_ports vxlan2
 658         bridge_stp off
 659         bridge_fd 0
 660         ip-forward on
 661         ip6-forward on
 662         arp-accept on
 663
 664
 665 auto vxlan3
 666 iface vxlan3 inet manual
 667         vxlan-id 3
 668         vxlan-local-tunnelip 192.168.0.2
 669         bridge-learning off
 670         bridge-arp-nd-suppress on
 671         bridge-unicast-flood off
 672         bridge-multicast-flood off
 673
 674
 675 auto vmbr3
 676 iface vmbr3 inet static
 677         address 10.0.3.254
 678         netmask 255.255.255.0
 679         hwaddress 44:39:39:FF:40:94
 680         bridge_ports vxlan3
 681         bridge_stp off
 682         bridge_fd 0
 683         ip-forward on
 684         ip6-forward on
 685         arp-accept on
 686 ----
 687
 688
 689 frr.conf
 690
 691 ----
 692 router bgp 1234
 693  bgp router-id 192.168.0.2
 694  no bgp default ipv4-unicast
 695  coalesce-time 1000
 696  neighbor 192.168.0.1 remote-as 1234
 697  neighbor 192.168.0.3 remote-as 1234
 698  !
 699  address-family l2vpn evpn
 700   neighbor 192.168.0.1 activate
 701   neighbor 192.168.0.3 activate
 702   advertise-all-vni
 703  exit-address-family
 704 !
 705 line vty
 706 !
 707 ----
 708
 709
 710 * node3
 711
 712 ----
 713 auto eno1
 714 iface eno1 inet manual
 715         mtu 1550
 716
 717 auto vmbr0
 718 iface vmbr0 inet static
 719          address 192.168.0.3
 720          netmask  255.255.255.0
 721          bridge_ports eno1
 722          bridge_stp off
 723          bridge_fd 0
 724
 725 auto vxlan2
 726 iface vxlan2 inet manual
 727         vxlan-id 2
 728         vxlan-local-tunnelip 192.168.0.3
 729         bridge-learning off
 730         bridge-arp-nd-suppress on
 731         bridge-unicast-flood off
 732         bridge-multicast-flood off
 733
 734
 735 auto vmbr2
 736 iface vmbr2 inet static
 737         address 10.0.2.254
 738         netmask 255.255.255.0
 739         hwaddress 44:39:39:FF:40:94
 740         bridge_ports vxlan2
 741         bridge_stp off
 742         bridge_fd 0
 743         ip-forward on
 744         ip6-forward on
 745         arp-accept on
 746
 747 auto vxlan3
 748 iface vxlan3 inet manual
 749         vxlan-id 3
 750         vxlan-local-tunnelip 192.168.0.3
 751         bridge-learning off
 752         bridge-arp-nd-suppress on
 753         bridge-unicast-flood off
 754         bridge-multicast-flood off
 755
 756 auto vmbr3
 757 iface vmbr3 inet static
 758         address 10.0.3.254
 759         netmask 255.255.255.0
 760         hwaddress 44:39:39:FF:40:94
 761         bridge_ports vxlan3
 762         bridge_stp off
 763         bridge_fd 0
 764         ip-forward on
 765         ip6-forward on
 766         arp-accept on
 767 ----
 768
 769
 770 frr.conf
 771
 772 ----
 773 router bgp 1234
 774  bgp router-id 192.168.0.3
 775  no bgp default ipv4-unicast
 776  coalesce-time 1000
 777  neighbor 192.168.0.1 remote-as 1234
 778  neighbor 192.168.0.2 remote-as 1234
 779  !
 780  address-family l2vpn evpn
 781   neighbor 192.168.0.1 activate
 782   neighbor 192.168.0.2 activate
 783   advertise-all-vni
 784  exit-address-family
 785 !
 786 line vty
 787 !
 788 ----
 789
 790
 791 symmetric model
 792 ^^^^^^^^^^^^^^^
 793
 794 With this model, you don't need to have all vxlan on all nodes.
 795 This model will also be needed to route traffic to an external router.
 796
 797 The symmetric model routes and bridges on both the ingress and the egress leafs.
 798 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 799 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 800 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 801 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 802
 803 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 804
 805 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 806
 807 * node1
 808
 809 ----
 810 auto vrf1
 811 iface vrf1
 812     vrf-table auto
 813
 814 auto eno1
 815 iface eno1 inet manual
 816         mtu 1550
 817
 818 auto vmbr0
 819 iface vmbr0 inet static
 820          address 192.168.0.1
 821          netmask  255.255.255.0
 822          bridge_ports eno1
 823          bridge_stp off
 824          bridge_fd 0
 825
 826 auto vxlan2
 827 iface vxlan2 inet manual
 828         vxlan-id 2
 829         vxlan-local-tunnelip 192.168.0.1
 830         bridge-learning off
 831         bridge-arp-nd-suppress on
 832         bridge-unicast-flood off
 833         bridge-multicast-flood off
 834
 835 auto vmbr2
 836 iface vmbr2 inet static
 837         bridge_ports vxlan2
 838         bridge_stp off
 839         bridge_fd 0
 840         address 10.0.2.254
 841         netmask 255.255.255.0
 842         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 843         vrf vrf1
 844         ip-forward on
 845         ip6-forward on
 846         arp-accept on
 847
 848 auto vxlan3
 849 iface vxlan3 inet manual
 850         vxlan-id 3
 851         vxlan-local-tunnelip 192.168.0.1
 852         bridge-learning off
 853         bridge-arp-nd-suppress on
 854         bridge-unicast-flood off
 855         bridge-multicast-flood off
 856
 857 auto vmbr3
 858 iface vmbr3 inet static
 859         bridge_ports vxlan3
 860         bridge_stp off
 861         bridge_fd 0
 862         address 10.0.3.254
 863         netmask 255.255.255.0
 864         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 865         vrf vrf1
 866         ip-forward on
 867         ip6-forward on
 868         arp-accept on
 869
 870 #interconnect vxlan-vfr l3vni
 871 auto vxlan4000
 872 iface vxlan4000 inet manual
 873         vxlan-id 4000
 874         vxlan-local-tunnelip 192.168.0.1
 875         bridge-learning off
 876         bridge-arp-nd-suppress on
 877         bridge-unicast-flood off
 878         bridge-multicast-flood off
 879
 880
 881 auto vmbr4000
 882 iface vmbr4000 inet manual
 883         bridge_ports vxlan4000
 884         bridge_stp off
 885         bridge_fd 0
 886         vrf vrf1
 887 ----
 888
 889 frr.conf
 890
 891 ----
 892 vrf vrf1
 893  vni 4000
 894  exit-vrf
 895 !
 896 router bgp 1234
 897  bgp router-id 192.168.0.1
 898  no bgp default ipv4-unicast
 899  coalesce-time 1000
 900  neighbor 192.168.0.2 remote-as 1234
 901  neighbor 192.168.0.3 remote-as 1234
 902  !
 903  address-family l2vpn evpn
 904   neighbor 192.168.0.2 activate
 905   neighbor 192.168.0.3 activate
 906   advertise-all-vni
 907  exit-address-family
 908 !
 909 line vty
 910 !
 911 ----
 912
 913
 914 * node2
 915
 916 ----
 917 auto vrf1
 918 iface vrf1
 919     vrf-table auto
 920
 921 auto eno1
 922 iface eno1 inet manual
 923         mtu 1550
 924
 925 auto vmbr0
 926 iface vmbr0 inet static
 927          address 192.168.0.2
 928          netmask  255.255.255.0
 929          bridge_ports eno1
 930          bridge_stp off
 931          bridge_fd 0
 932
 933 auto vxlan2
 934 iface vxlan2 inet manual
 935         vxlan-id 2
 936         vxlan-local-tunnelip 192.168.0.2
 937         bridge-learning off
 938         bridge-arp-nd-suppress on
 939         bridge-unicast-flood off
 940         bridge-multicast-flood off
 941
 942 auto vmbr2
 943 iface vmbr2 inet static
 944         bridge_ports vxlan2
 945         bridge_stp off
 946         bridge_fd 0
 947         address 10.0.2.254
 948         netmask 255.255.255.0
 949         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 950         vrf vrf1
 951         ip-forward on
 952         ip6-forward on
 953         arp-accept on
 954
 955 auto vxlan3
 956 iface vxlan3 inet manual
 957         vxlan-id 3
 958         vxlan-local-tunnelip 192.168.0.2
 959         bridge-learning off
 960         bridge-arp-nd-suppress on
 961         bridge-unicast-flood off
 962         bridge-multicast-flood off
 963
 964 auto vmbr3
 965 iface vmbr3 inet static
 966         bridge_ports vxlan3
 967         bridge_stp off
 968         bridge_fd 0
 969         address 10.0.3.254
 970         netmask 255.255.255.0
 971         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 972         vrf vrf1
 973         ip-forward on
 974         ip6-forward on
 975         arp-accept on
 976
 977 #interconnect vxlan-vfr l3vni
 978 auto vxlan4000
 979 iface vxlan4000 inet manual
 980         vxlan-id 4000
 981         vxlan-local-tunnelip 192.168.0.2
 982         bridge-learning off
 983         bridge-arp-nd-suppress on
 984         bridge-unicast-flood off
 985         bridge-multicast-flood off
 986
 987
 988 auto vmbr4000
 989 iface vmbr4000 inet manual
 990         bridge_ports vxlan4000
 991         bridge_stp off
 992         bridge_fd 0
 993         vrf vrf1
 994 ----
 995
 996
 997 frr.conf
 998
 999 ----
1000 vrf vrf1
1001  vni 4000
1002  exit-vrf
1003 !
1004 router bgp 1234
1005  bgp router-id 192.168.0.2
1006  no bgp default ipv4-unicast
1007  coalesce-time 1000
1008  neighbor 192.168.0.1 remote-as 1234
1009  neighbor 192.168.0.3 remote-as 1234
1010  !
1011  address-family l2vpn evpn
1012   neighbor 192.168.0.1 activate
1013   neighbor 192.168.0.3 activate
1014   advertise-all-vni
1015  exit-address-family
1016 !
1017 line vty
1018 !
1019 ----
1020
1021
1022 * node3
1023
1024 ----
1025 auto vrf1
1026 iface vrf1
1027     vrf-table auto
1028
1029 auto eno1
1030 iface eno1 inet manual
1031         mtu 1550
1032
1033 auto vmbr0
1034 iface vmbr0 inet static
1035          address 192.168.0.3
1036          netmask  255.255.255.0
1037          bridge_ports eno1
1038          bridge_stp off
1039          bridge_fd 0
1040
1041 auto vxlan2
1042 iface vxlan2 inet manual
1043         vxlan-id 2
1044         vxlan-local-tunnelip 192.168.0.3
1045         bridge-learning off
1046         bridge-arp-nd-suppress on
1047         bridge-unicast-flood off
1048         bridge-multicast-flood off
1049
1050 auto vmbr2
1051 iface vmbr2 inet static
1052         bridge_ports vxlan2
1053         bridge_stp off
1054         bridge_fd 0
1055         address 10.0.2.254
1056         netmask 255.255.255.0
1057         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1058         vrf vrf1
1059         ip-forward on
1060         ip6-forward on
1061         arp-accept on
1062
1063 auto vxlan3
1064 iface vxlan3 inet manual
1065         vxlan-id 3
1066         vxlan-local-tunnelip 192.168.0.3
1067         bridge-learning off
1068         bridge-arp-nd-suppress on
1069         bridge-unicast-flood off
1070         bridge-multicast-flood off
1071
1072 auto vmbr3
1073 iface vmbr3 inet static
1074         bridge_ports vxlan3
1075         bridge_stp off
1076         bridge_fd 0
1077         address 10.0.3.254
1078         netmask 255.255.255.0
1079         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1080         vrf vrf1
1081         ip-forward on
1082         ip6-forward on
1083         arp-accept on
1084
1085 #interconnect vxlan-vfr l3vni
1086 auto vxlan4000
1087 iface vxlan4000 inet manual
1088         vxlan-id 4000
1089         vxlan-local-tunnelip 192.168.0.3
1090         bridge-learning off
1091         bridge-arp-nd-suppress on
1092         bridge-unicast-flood off
1093         bridge-multicast-flood off
1094
1095
1096 auto vmbr4000
1097 iface vmbr4000 inet manual
1098         bridge_ports vxlan4000
1099         bridge_stp off
1100         bridge_fd 0
1101         vrf vrf1
1102 ----
1103
1104
1105 frr.conf
1106
1107 ----
1108 vrf vrf1
1109  vni 4000
1110  exit-vrf
1111 !
1112 router bgp 1234
1113  bgp router-id 192.168.0.3
1114  no bgp default ipv4-unicast
1115  coalesce-time 1000
1116  neighbor 192.168.0.1 remote-as 1234
1117  neighbor 192.168.0.2 remote-as 1234
1118  !
1119  address-family l2vpn evpn
1120   neighbor 192.168.0.1 activate
1121   neighbor 192.168.0.2 activate
1122   advertise-all-vni
1123  exit-address-family
1124 !
1125 line vty
1126 !
1127 ----
1128
1129 VXLAN layer3 routing with anycast gateway + routing to outside with external router with static default gw
1130 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1131 Routing to outside need the symmetric model.
1132
1133 1 gateway node
1134 ^^^^^^^^^^^^^^
1135 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1136 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1137
1138
1139 *node1
1140
1141 ----
1142 auto vrf1
1143 iface vrf1
1144     vrf-table auto
1145
1146 auto eno1
1147 iface eno1 inet manual
1148         mtu 1550
1149
1150 auto vmbr0
1151 iface vmbr0 inet static
1152          address 192.168.0.1
1153          netmask  255.255.255.0
1154          gateway 192.168.0.254
1155          bridge_ports eno1
1156          bridge_stp off
1157          bridge_fd 0
1158          ip-forward on
1159          ip6-forward on
1160
1161 auto vxlan2
1162 iface vxlan2 inet manual
1163         vxlan-id 2
1164         vxlan-local-tunnelip 192.168.0.1
1165         bridge-learning off
1166         bridge-arp-nd-suppress on
1167         bridge-unicast-flood off
1168         bridge-multicast-flood off
1169
1170 auto vmbr2
1171 iface vmbr2 inet static
1172         bridge_ports vxlan2
1173         bridge_stp off
1174         bridge_fd 0
1175         address 10.0.2.254
1176         netmask 255.255.255.0
1177         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1178         vrf vrf1
1179         ip-forward on
1180         ip6-forward on
1181         arp-accept on
1182
1183 auto vxlan3
1184 iface vxlan3 inet manual
1185         vxlan-id 3
1186         vxlan-local-tunnelip 192.168.0.1
1187         bridge-learning off
1188         bridge-arp-nd-suppress on
1189         bridge-unicast-flood off
1190         bridge-multicast-flood off
1191
1192 auto vmbr3
1193 iface vmbr3 inet static
1194         bridge_ports vxlan3
1195         bridge_stp off
1196         bridge_fd 0
1197         address 10.0.3.254
1198         netmask 255.255.255.0
1199         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1200         vrf vrf1
1201         ip-forward on
1202         ip6-forward on
1203         arp-accept on
1204
1205 #interconnect vxlan-vfr l3vni
1206 auto vxlan4000
1207 iface vxlan4000 inet manual
1208         vxlan-id 4000
1209         vxlan-local-tunnelip 192.168.0.1
1210         bridge-learning off
1211         bridge-arp-nd-suppress on
1212         bridge-unicast-flood off
1213         bridge-multicast-flood off
1214
1215 auto vmbr4000
1216 iface vmbr4000 inet manual
1217         bridge_ports vxlan4000
1218         bridge_stp off
1219         bridge_fd 0
1220         vrf vrf1
1221 ----
1222
1223
1224 frr.conf
1225
1226 ----
1227 vrf vrf1
1228  vni 4000
1229  exit-vrf
1230 !
1231 router bgp 1234
1232  bgp router-id 192.168.0.1
1233  no bgp default ipv4-unicast
1234  coalesce-time 1000
1235  neighbor 192.168.0.2 remote-as 1234
1236  neighbor 192.168.0.3 remote-as 1234
1237  !
1238  address-family ipv4 unicast
1239   import vrf vrf1
1240  exit-address-family
1241  !
1242  address-family ipv6 unicast
1243   import vrf vrf1
1244  exit-address-family
1245  !
1246  address-family l2vpn evpn
1247   neighbor 192.168.0.2 activate
1248   neighbor 192.168.0.3 activate
1249   advertise-all-vni
1250  exit-address-family
1251 !
1252 router bgp 1234 vrf vrf1
1253 !
1254  address-family ipv4 unicast
1255   redistribute connected
1256  exit-address-family
1257  !
1258  address-family ipv6 unicast
1259   redistribute connected
1260  exit-address-family
1261  !
1262  address-family l2vpn evpn
1263   default-originate ipv4
1264   default-originate ipv6
1265  exit-address-family
1266 !
1267 line vty
1268 !
1269 ----
1270
1271
1272 * node2
1273
1274 ----
1275 auto vrf1
1276 iface vrf1
1277     vrf-table auto
1278
1279 auto eno1
1280 iface eno1 inet manual
1281         mtu 1550
1282
1283 auto vmbr0
1284 iface vmbr0 inet static
1285          address 192.168.0.2
1286          netmask  255.255.255.0
1287          bridge_ports eno1
1288          bridge_stp off
1289          bridge_fd 0
1290
1291 auto vxlan2
1292 iface vxlan2 inet manual
1293         vxlan-id 2
1294         vxlan-local-tunnelip 192.168.0.2
1295         bridge-learning off
1296         bridge-arp-nd-suppress on
1297         bridge-unicast-flood off
1298         bridge-multicast-flood off
1299
1300 auto vmbr2
1301 iface vmbr2 inet static
1302         bridge_ports vxlan2
1303         bridge_stp off
1304         bridge_fd 0
1305         address 10.0.2.254
1306         netmask 255.255.255.0
1307         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1308         vrf vrf1
1309         ip-forward on
1310         ip6-forward on
1311         arp-accept on
1312
1313 auto vxlan3
1314 iface vxlan3 inet manual
1315         vxlan-id 3
1316         vxlan-local-tunnelip 192.168.0.2
1317         bridge-learning off
1318         bridge-arp-nd-suppress on
1319         bridge-unicast-flood off
1320         bridge-multicast-flood off
1321
1322 auto vmbr3
1323 iface vmbr3 inet static
1324         bridge_ports vxlan3
1325         bridge_stp off
1326         bridge_fd 0
1327         address 10.0.3.254
1328         netmask 255.255.255.0
1329         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1330         vrf vrf1
1331         ip-forward on
1332         ip6-forward on
1333         arp-accept on
1334
1335 #interconnect vxlan-vfr l3vni
1336 auto vxlan4000
1337 iface vxlan4000 inet manual
1338         vxlan-id 4000
1339         vxlan-local-tunnelip 192.168.0.2
1340         bridge-learning off
1341         bridge-arp-nd-suppress on
1342         bridge-unicast-flood off
1343         bridge-multicast-flood off
1344
1345
1346 auto vmbr4000
1347 iface vmbr4000 inet manual
1348         bridge_ports vxlan4000
1349         bridge_stp off
1350         bridge_fd 0
1351         vrf vrf1
1352 ----
1353
1354
1355 frr.conf
1356
1357 ----
1358 vrf vrf1
1359  vni 4000
1360  exit-vrf
1361 !
1362 router bgp 1234
1363  bgp router-id 192.168.0.2
1364  no bgp default ipv4-unicast
1365  coalesce-time 1000
1366  neighbor 192.168.0.1 remote-as 1234
1367  neighbor 192.168.0.3 remote-as 1234
1368  !
1369  address-family l2vpn evpn
1370   neighbor 192.168.0.1 activate
1371   neighbor 192.168.0.3 activate
1372   advertise-all-vni
1373  exit-address-family
1374 !
1375 line vty
1376 !
1377 ----
1378
1379
1380 * node3
1381
1382 ----
1383 auto vrf1
1384 iface vrf1
1385     vrf-table auto
1386
1387 auto eno1
1388 iface eno1 inet manual
1389         mtu 1550
1390
1391 auto vmbr0
1392 iface vmbr0 inet static
1393          address 192.168.0.3
1394          netmask  255.255.255.0
1395          bridge_ports eno1
1396          bridge_stp off
1397          bridge_fd 0
1398
1399 auto vxlan2
1400 iface vxlan2 inet manual
1401         vxlan-id 2
1402         vxlan-local-tunnelip 192.168.0.3
1403         bridge-learning off
1404         bridge-arp-nd-suppress on
1405         bridge-unicast-flood off
1406         bridge-multicast-flood off
1407
1408 auto vmbr2
1409 iface vmbr2 inet static
1410         bridge_ports vxlan2
1411         bridge_stp off
1412         bridge_fd 0
1413         address 10.0.2.254
1414         netmask 255.255.255.0
1415         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1416         vrf vrf1
1417         ip-forward on
1418         ip6-forward on
1419         arp-accept on
1420
1421 auto vxlan3
1422 iface vxlan3 inet manual
1423         vxlan-id 3
1424         vxlan-local-tunnelip 192.168.0.3
1425         bridge-learning off
1426         bridge-arp-nd-suppress on
1427         bridge-unicast-flood off
1428         bridge-multicast-flood off
1429
1430 auto vmbr3
1431 iface vmbr3 inet static
1432         bridge_ports vxlan3
1433         bridge_stp off
1434         bridge_fd 0
1435         address 10.0.3.254
1436         netmask 255.255.255.0
1437         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1438         vrf vrf1
1439         ip-forward on
1440         ip6-forward on
1441         arp-accept on
1442
1443 #interconnect vxlan-vfr l3vni
1444 auto vxlan4000
1445 iface vxlan4000 inet manual
1446         vxlan-id 4000
1447         vxlan-local-tunnelip 192.168.0.3
1448         bridge-learning off
1449         bridge-arp-nd-suppress on
1450         bridge-unicast-flood off
1451         bridge-multicast-flood off
1452
1453
1454 auto vmbr4000
1455 iface vmbr4000 inet manual
1456         bridge_ports vxlan4000
1457         bridge_stp off
1458         bridge_fd 0
1459         vrf vrf1
1460 ----
1461
1462
1463 frr.conf
1464
1465 ----
1466 vrf vrf1
1467  vni 4000
1468  exit-vrf
1469 !
1470 router bgp 1234
1471  bgp router-id 192.168.0.3
1472  no bgp default ipv4-unicast
1473  coalesce-time 1000
1474  neighbor 192.168.0.1 remote-as 1234
1475  neighbor 192.168.0.2 remote-as 1234
1476  !
1477  address-family l2vpn evpn
1478   neighbor 192.168.0.1 activate
1479   neighbor 192.168.0.2 activate
1480   advertise-all-vni
1481  exit-address-family
1482 !
1483 line vty
1484 !
1485 ----
1486
1487 multiple gateway nodes
1488 ^^^^^^^^^^^^^^^^^^^^^^
1489 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1490 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1491 and announce this default gw in the vrf (default originate)
1492 The external router have ecmp routes to all proxmox nodes.(balancing).
1493 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1494 vxlan the packet to final destination.
1495
1496 If you have multiple gateway nodes, disable rp_filter as packet could incoming in a 1 node, and outgoing
1497 to another node.
1498
1499 sysctl.conf tuning
1500 -----
1501 net.ipv4.conf.default.rp_filter=0
1502 net.ipv4.conf.all.rp_filter=0
1503 -----
1504
1505
1506 *node1
1507
1508 ----
1509 auto vrf1
1510 iface vrf1
1511     vrf-table auto
1512
1513 auto eno1
1514 iface eno1 inet manual
1515         mtu 1550
1516
1517 auto vmbr0
1518 iface vmbr0 inet static
1519          address 192.168.0.1
1520          netmask  255.255.255.0
1521          gateway 192.168.0.254
1522          bridge_ports eno1
1523          bridge_stp off
1524          bridge_fd 0
1525          ip-forward on
1526          ip6-forward on
1527
1528 auto vxlan2
1529 iface vxlan2 inet manual
1530         vxlan-id 2
1531         vxlan-local-tunnelip 192.168.0.1
1532         bridge-learning off
1533         bridge-arp-nd-suppress on
1534         bridge-unicast-flood off
1535         bridge-multicast-flood off
1536
1537 auto vmbr2
1538 iface vmbr2 inet static
1539         bridge_ports vxlan2
1540         bridge_stp off
1541         bridge_fd 0
1542         address 10.0.2.254
1543         netmask 255.255.255.0
1544         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1545         vrf vrf1
1546         ip-forward on
1547         ip6-forward on
1548         arp-accept on
1549
1550 auto vxlan3
1551 iface vxlan3 inet manual
1552         vxlan-id 3
1553         vxlan-local-tunnelip 192.168.0.1
1554         bridge-learning off
1555         bridge-arp-nd-suppress on
1556         bridge-unicast-flood off
1557         bridge-multicast-flood off
1558
1559 auto vmbr3
1560 iface vmbr3 inet static
1561         bridge_ports vxlan3
1562         bridge_stp off
1563         bridge_fd 0
1564         address 10.0.3.254
1565         netmask 255.255.255.0
1566         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1567         vrf vrf1
1568         ip-forward on
1569         ip6-forward on
1570         arp-accept on
1571
1572 #interconnect vxlan-vfr l3vni
1573 auto vxlan4000
1574 iface vxlan4000 inet manual
1575         vxlan-id 4000
1576         vxlan-local-tunnelip 192.168.0.1
1577         bridge-learning off
1578         bridge-arp-nd-suppress on
1579         bridge-unicast-flood off
1580         bridge-multicast-flood off
1581
1582 auto vmbr4000
1583 iface vmbr4000 inet manual
1584         bridge_ports vxlan4000
1585         bridge_stp off
1586         bridge_fd 0
1587         vrf vrf1
1588 ----
1589
1590
1591 frr.conf
1592
1593 ----
1594 vrf vrf1
1595  vni 4000
1596  exit-vrf
1597 !
1598 router bgp 1234
1599  bgp router-id 192.168.0.1
1600  no bgp default ipv4-unicast
1601  coalesce-time 1000
1602  neighbor 192.168.0.2 remote-as 1234
1603  neighbor 192.168.0.3 remote-as 1234
1604  !
1605  address-family ipv4 unicast
1606   import vrf vrf1
1607  exit-address-family
1608  !
1609  address-family ipv6 unicast
1610   import vrf vrf1
1611  exit-address-family
1612  !
1613  address-family l2vpn evpn
1614   neighbor 192.168.0.2 activate
1615   neighbor 192.168.0.3 activate
1616   advertise-all-vni
1617  exit-address-family
1618 !
1619 router bgp 1234 vrf vrf1
1620 !
1621  address-family ipv4 unicast
1622   redistribute connected
1623  exit-address-family
1624  !
1625  address-family ipv6 unicast
1626   redistribute connected
1627  exit-address-family
1628  !
1629  address-family l2vpn evpn
1630   default-originate ipv4
1631   default-originate ipv6
1632  exit-address-family
1633 !
1634 line vty
1635 !
1636 ----
1637
1638
1639 * node2
1640
1641 ----
1642 auto vrf1
1643 iface vrf1
1644     vrf-table auto
1645
1646 auto eno1
1647 iface eno1 inet manual
1648         mtu 1550
1649
1650 auto vmbr0
1651 iface vmbr0 inet static
1652          address 192.168.0.2
1653          netmask  255.255.255.0
1654          gateway 192.168.0.254
1655          bridge_ports eno1
1656          bridge_stp off
1657          bridge_fd 0
1658          ip-forward on
1659          ip6-forward on
1660
1661 auto vxlan2
1662 iface vxlan2 inet manual
1663         vxlan-id 2
1664         vxlan-local-tunnelip 192.168.0.2
1665         bridge-learning off
1666         bridge-arp-nd-suppress on
1667         bridge-unicast-flood off
1668         bridge-multicast-flood off
1669
1670 auto vmbr2
1671 iface vmbr2 inet static
1672         bridge_ports vxlan2
1673         bridge_stp off
1674         bridge_fd 0
1675         address 10.0.2.254
1676         netmask 255.255.255.0
1677         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1678         vrf vrf1
1679         ip-forward on
1680         ip6-forward on
1681         arp-accept on
1682
1683 auto vxlan3
1684 iface vxlan3 inet manual
1685         vxlan-id 3
1686         vxlan-local-tunnelip 192.168.0.2
1687         bridge-learning off
1688         bridge-arp-nd-suppress on
1689         bridge-unicast-flood off
1690         bridge-multicast-flood off
1691
1692 auto vmbr3
1693 iface vmbr3 inet static
1694         bridge_ports vxlan3
1695         bridge_stp off
1696         bridge_fd 0
1697         address 10.0.3.254
1698         netmask 255.255.255.0
1699         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1700         vrf vrf1
1701         ip-forward on
1702         ip6-forward on
1703         arp-accept on
1704
1705 #interconnect vxlan-vfr l3vni
1706 auto vxlan4000
1707 iface vxlan4000 inet manual
1708         vxlan-id 4000
1709         vxlan-local-tunnelip 192.168.0.2
1710         bridge-learning off
1711         bridge-arp-nd-suppress on
1712         bridge-unicast-flood off
1713         bridge-multicast-flood off
1714
1715
1716 auto vmbr4000
1717 iface vmbr4000 inet manual
1718         bridge_ports vxlan4000
1719         bridge_stp off
1720         bridge_fd 0
1721         vrf vrf1
1722 ----
1723
1724
1725 frr.conf
1726
1727 ----
1728 vrf vrf1
1729  vni 4000
1730  exit-vrf
1731 !
1732 router bgp 1234
1733  bgp router-id 192.168.0.2
1734  no bgp default ipv4-unicast
1735  coalesce-time 1000
1736  neighbor 192.168.0.1 remote-as 1234
1737  neighbor 192.168.0.3 remote-as 1234
1738  !
1739  address-family ipv4 unicast
1740   import vrf vrf1
1741  exit-address-family
1742  !
1743  address-family ipv6 unicast
1744   import vrf vrf1
1745  exit-address-family
1746  !
1747  address-family l2vpn evpn
1748   neighbor 192.168.0.1 activate
1749   neighbor 192.168.0.3 activate
1750   advertise-all-vni
1751  exit-address-family
1752 !
1753  address-family ipv4 unicast
1754   redistribute connected
1755  exit-address-family
1756  !
1757  address-family ipv6 unicast
1758   redistribute connected
1759  exit-address-family
1760  !
1761  address-family l2vpn evpn
1762   default-originate ipv4
1763   default-originate ipv6
1764  exit-address-family
1765 !
1766 line vty
1767 !
1768 ----
1769
1770
1771 * node3
1772
1773 ----
1774 auto vrf1
1775 iface vrf1
1776     vrf-table auto
1777
1778 auto eno1
1779 iface eno1 inet manual
1780         mtu 1550
1781
1782 auto vmbr0
1783 iface vmbr0 inet static
1784          address 192.168.0.3
1785          netmask  255.255.255.0
1786          gateway 192.168.0.254
1787          bridge_ports eno1
1788          bridge_stp off
1789          bridge_fd 0
1790          ip-forward on
1791          ip6-forward on
1792
1793 auto vxlan2
1794 iface vxlan2 inet manual
1795         vxlan-id 2
1796         vxlan-local-tunnelip 192.168.0.3
1797         bridge-learning off
1798         bridge-arp-nd-suppress on
1799         bridge-unicast-flood off
1800         bridge-multicast-flood off
1801
1802 auto vmbr2
1803 iface vmbr2 inet static
1804         bridge_ports vxlan2
1805         bridge_stp off
1806         bridge_fd 0
1807         address 10.0.2.254
1808         netmask 255.255.255.0
1809         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1810         vrf vrf1
1811         ip-forward on
1812         ip6-forward on
1813         arp-accept on
1814
1815 auto vxlan3
1816 iface vxlan3 inet manual
1817         vxlan-id 3
1818         vxlan-local-tunnelip 192.168.0.3
1819         bridge-learning off
1820         bridge-arp-nd-suppress on
1821         bridge-unicast-flood off
1822         bridge-multicast-flood off
1823
1824 auto vmbr3
1825 iface vmbr3 inet static
1826         bridge_ports vxlan3
1827         bridge_stp off
1828         bridge_fd 0
1829         address 10.0.3.254
1830         netmask 255.255.255.0
1831         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1832         vrf vrf1
1833         ip-forward on
1834         ip6-forward on
1835         arp-accept on
1836
1837 #interconnect vxlan-vfr l3vni
1838 auto vxlan4000
1839 iface vxlan4000 inet manual
1840         vxlan-id 4000
1841         vxlan-local-tunnelip 192.168.0.3
1842         bridge-learning off
1843         bridge-arp-nd-suppress on
1844         bridge-unicast-flood off
1845         bridge-multicast-flood off
1846
1847
1848 auto vmbr4000
1849 iface vmbr4000 inet manual
1850         bridge_ports vxlan4000
1851         bridge_stp off
1852         bridge_fd 0
1853         vrf vrf1
1854 ----
1855
1856
1857 frr.conf
1858
1859 ----
1860 vrf vrf1
1861  vni 4000
1862  exit-vrf
1863 !
1864 router bgp 1234
1865  bgp router-id 192.168.0.3
1866  no bgp default ipv4-unicast
1867  coalesce-time 1000
1868  neighbor 192.168.0.1 remote-as 1234
1869  neighbor 192.168.0.2 remote-as 1234
1870  !
1871  address-family ipv4 unicast
1872   import vrf vrf1
1873  exit-address-family
1874  !
1875  address-family ipv6 unicast
1876   import vrf vrf1
1877  exit-address-family
1878  !
1879  address-family l2vpn evpn
1880   neighbor 192.168.0.1 activate
1881   neighbor 192.168.0.2 activate
1882   advertise-all-vni
1883  exit-address-family
1884 !
1885 router bgp 1234 vrf vrf1
1886 !
1887  address-family ipv4 unicast
1888   redistribute connected
1889  exit-address-family
1890  !
1891  address-family ipv6 unicast
1892   redistribute connected
1893  exit-address-family
1894  !
1895  address-family l2vpn evpn
1896   default-originate ipv4
1897   default-originate ipv6
1898  exit-address-family
1899 !
1900 line vty
1901 !
1902 ----
1903
1904 Note
1905 ^^^^
1906
1907 If your external router doesn't support 'ECMP static routes' to reach multiple
1908 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1909 Virtual Router Redundancy Protocol (VRRP).
1910
1911 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1912 Node1 is the primary with failover to node2 in case of outage.
1913
1914 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1915 #TODO : It should be possible to do it with frr directly with last version.
1916
1917 * node1
1918
1919 ----
1920 auto vmbr0
1921 iface vmbr0 inet static
1922          address 192.168.0.1
1923          netmask  255.255.255.0
1924          gateway 192.168.0.254
1925          bridge_ports eno1
1926          bridge_stp off
1927          bridge_fd 0
1928          vrrp-id 1
1929          vrrp-priority 1
1930          vrrp-virtual-ip 192.168.0.10
1931 ----
1932
1933 * node2
1934
1935 ----
1936 auto vmbr0
1937 iface vmbr0 inet static
1938          address 192.168.0.2
1939          netmask  255.255.255.0
1940          gateway 192.168.0.254
1941          bridge_ports eno1
1942          bridge_stp off
1943          bridge_fd 0
1944          vrrp-id 1
1945          vrrp-priority 2
1946          vrrp-virtual-ip 192.168.0.10
1947 ----
1948
1949
1950
1951 gateway node(s) with a upstream bgp router
1952 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1953 Setup is almost the same than with a static gateway, but we'll connect to an upstream bgp router.
1954
1955 example with node1 as gateway (192.168.0.1) for evpn-bgp, and an upstream bgp router (running frr too) 192.168.0.254.
1956
1957 * node1
1958
1959 frr.conf
1960 ----
1961 vrf vrf1
1962  vni 4000
1963  exit-vrf
1964 !
1965 router bgp 1234
1966  bgp router-id 192.168.0.1
1967  no bgp default ipv4-unicast
1968  coalesce-time 1000
1969  neighbor 192.168.0.2 remote-as 1234
1970  neighbor 192.168.0.3 remote-as 1234
1971  neighbor 192.168.0.254 remote-as external
1972  !
1973  address-family ipv4 unicast
1974   import vrf vrf1
1975   neighbor 192.168.0.254 activate
1976  exit-address-family
1977  !
1978  address-family ipv6 unicast
1979   import vrf vrf1
1980   neighbor 192.168.0.254 activate
1981  exit-address-family
1982  !
1983  address-family l2vpn evpn
1984   neighbor 192.168.0.1 activate
1985   neighbor 192.168.0.2 activate
1986   neighbor 192.168.0.254 activate
1987   advertise-all-vni
1988  exit-address-family
1989 !
1990 router bgp 1234 vrf vrf1
1991 !
1992  address-family ipv4 unicast
1993   redistribute connected
1994  exit-address-family
1995  !
1996  address-family ipv6 unicast
1997   redistribute connected
1998  exit-address-family
1999  !
2000  address-family l2vpn evpn
2001   default-originate ipv4
2002   default-originate ipv6
2003  exit-address-family
2004 !
2005 line vty
2006 !
2007 ----
2008
2009 * bgp router
2010
2011 frr.conf
2012 ----
2013 ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24
2014 ip prefix-list NO32 seq 20 deny any
2015 !
2016 router bgp 25253
2017  bgp router-id 192.168.0.254
2018  bgp bestpath as-path multipath-relax
2019  neighbor 192.168.0.1 remote-as external
2020  neighbor 192.168.0.1 capability extended-nexthop
2021  !
2022  address-family ipv4 unicast
2023   neighbor 192.168.0.1 default-originate
2024   neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
2025  exit-address-family
2026  !
2027  address-family ipv6 unicast
2028   neighbor 192.168.0.1 default-originate
2029   neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
2030  exit-address-family
2031  !
2032 !
2033 ---
2034
2035 Route Reflectors
2036 ^^^^^^^^^^^^^^^^
2037 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
2038 to avoid that all node peers with each others nodes.
2039 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
2040 single point of failure, a minimum of two servers acting as an RR is highly
2041 recommended for redundancy.
2042
2043 Below is an example of configuration with 'frr', with `rrserver1
2044 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
2045
2046 rrserver1
2047 ----
2048 router bgp 1234
2049   bgp router-id 192.168.0.200
2050   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
2051   bgp log-neighbor-changes
2052   no bgp default ipv4-unicast
2053   neighbor fabric peer-group
2054   neighbor fabric remote-as 1234
2055   neighbor fabric capability extended-nexthop
2056   neighbor fabric update-source 192.168.0.200
2057   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
2058   !
2059   address-family l2vpn evpn
2060    neighbor fabric activate
2061    neighbor fabric route-reflector-client
2062    neighbor fabric allowas-in
2063   exit-address-family
2064   !
2065   exit
2066 !
2067 ---
2068
2069 rrserver2
2070 ----
2071 router bgp 1234
2072   bgp router-id 192.168.0.201
2073   bgp cluster-id 1.1.1.1
2074   bgp log-neighbor-changes
2075   no bgp default ipv4-unicast
2076   neighbor fabric peer-group
2077   neighbor fabric remote-as 1234
2078   neighbor fabric capability extended-nexthop
2079   neighbor fabric update-source 192.168.0.201
2080   bgp listen range 192.168.0.0/24 peer-group fabric
2081   !
2082   address-family l2vpn evpn
2083    neighbor fabric activate
2084    neighbor fabric route-reflector-client
2085    neighbor fabric allowas-in
2086   exit-address-family
2087   !
2088   exit
2089 !
2090 ---
2091
2092 proxmoxnode(s)
2093 ----
2094 router bgp 1234
2095  bgp router-id 192.168.0.x
2096  no bgp default ipv4-unicast
2097  coalesce-time 1000
2098  neighbor 192.168.0.200 remote-as 1234
2099  neighbor 192.168.0.201 remote-as 1234
2100  !
2101  address-family l2vpn evpn
2102   neighbor 192.168.0.200 activate
2103   neighbor 192.168.0.201 activate
2104   advertise-all-vni
2105  exit-address-family
2106 !
2107 ----