vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  coalesce-time 1000
 371  neighbor 192.168.0.2 remote-as 1234
 372  neighbor 192.168.0.3 remote-as 1234
 373  !
 374  address-family l2vpn evpn
 375   neighbor 192.168.0.2 activate
 376   neighbor 192.168.0.3 activate
 377   advertise-all-vni
 378  exit-address-family
 379 !
 380 line vty
 381 !
 382 ----
 383
 384
 385 * node2
 386
 387 ----
 388 auto eno1
 389 iface eno1 inet manual
 390         mtu 1550
 391
 392 auto vmbr0
 393 iface vmbr0 inet static
 394         address 192.168.0.2
 395         netmask 255.255.255.0
 396         bridge_ports eno1
 397         bridge_stp off
 398         bridge_fd 0
 399
 400 auto vxlan2
 401 iface vxlan2 inet manual
 402         vxlan-id 2
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr2
 411 iface vmbr2 inet manual
 412         bridge_ports vxlan2
 413         bridge_stp off
 414         bridge_fd 0
 415
 416 auto vxlan3
 417 iface vxlan3 inet manual
 418         vxlan-id 3
 419         vxlan-local-tunnelip 192.168.0.2
 420         bridge-learning off
 421         bridge-arp-nd-suppress on
 422         bridge-unicast-flood off
 423         bridge-multicast-flood off
 424
 425
 426 auto vmbr3
 427 iface vmbr3 inet manual
 428         bridge_ports vxlan3
 429         bridge_stp off
 430         bridge_fd 0
 431 ----
 432
 433
 434 /etc/frr/frr.conf
 435
 436 ----
 437 router bgp 1234
 438  no bgp default ipv4-unicast
 439  coalesce-time 1000
 440  neighbor 192.168.0.1 remote-as 1234
 441  neighbor 192.168.0.3 remote-as 1234
 442  !
 443  address-family l2vpn evpn
 444   neighbor 192.168.0.1 activate
 445   neighbor 192.168.0.3 activate
 446   advertise-all-vni
 447  exit-address-family
 448 !
 449 line vty
 450 !
 451 ----
 452
 453
 454 * node3
 455
 456 ----
 457 auto eno1
 458 iface eno1 inet manual
 459         mtu 1550
 460
 461 auto vmbr0
 462 iface vmbr0 inet static
 463         address 192.168.0.2
 464         netmask 255.255.255.0
 465         bridge_ports eno1
 466         bridge_stp off
 467         bridge_fd 0
 468
 469 auto vxlan2
 470 iface vxlan2 inet manual
 471         vxlan-id 2
 472         vxlan-local-tunnelip 192.168.0.3
 473         bridge-learning off
 474         bridge-arp-nd-suppress on
 475         bridge-unicast-flood off
 476         bridge-multicast-flood off
 477
 478
 479 auto vmbr2
 480 iface vmbr2 inet manual
 481         bridge_ports vxlan2
 482         bridge_stp off
 483         bridge_fd 0
 484
 485 auto vxlan3
 486 iface vxlan3 inet manual
 487         vxlan-id 3
 488         vxlan-local-tunnelip 192.168.0.3
 489         bridge-learning off
 490         bridge-arp-nd-suppress on
 491         bridge-unicast-flood off
 492         bridge-multicast-flood off
 493
 494
 495 auto vmbr3
 496 iface vmbr3 inet manual
 497         bridge_ports vxlan3
 498         bridge_stp off
 499         bridge_fd 0
 500 ----
 501
 502
 503 /etc/frr/frr.conf
 504
 505
 506 ----
 507 router bgp 1234
 508  no bgp default ipv4-unicast
 509  coalesce-time 1000
 510  neighbor 192.168.0.1 remote-as 1234
 511  neighbor 192.168.0.2 remote-as 1234
 512  !
 513  address-family l2vpn evpn
 514   neighbor 192.168.0.1 activate
 515   neighbor 192.168.0.2 activate
 516   advertise-all-vni
 517  exit-address-family
 518 !
 519 line vty
 520 !
 521 ----
 522
 523 VXLAN layer3 routing with anycast gateway
 524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 525
 526 With this need, each vmbr bridge will be the gateway for the vm.
 527 Same vmbr on different node, will have same ip address and same mac address,
 528 to have working vm live migration and no network disruption.
 529
 530 VXLAN layer3 routing only work with FRR and non-aware bridge.
 531 (vlan aware bridge support is buggy currently).
 532
 533 asymmetric model
 534 ^^^^^^^^^^^^^^^^
 535
 536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 537
 538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 539 but only bridging on the egress.
 540 This results in bi-directional VXLAN traffic traveling on different VNIs
 541 in each direction (always the destination VNI) across the routed infrastructure.
 542
 543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 544
 545 * node1
 546
 547 ----
 548 auto eno1
 549 iface eno1 inet manual
 550         mtu 1550
 551
 552 auto vmbr0
 553 iface vmbr0 inet static
 554          address 192.168.0.1
 555          netmask  255.255.255.0
 556          bridge_ports eno1
 557          bridge_stp off
 558          bridge_fd 0
 559
 560 auto vxlan2
 561 iface vxlan2 inet manual
 562         vxlan-id 2
 563         vxlan-local-tunnelip 192.168.0.1
 564         bridge-learning off
 565         bridge-arp-nd-suppress on
 566         bridge-unicast-flood off
 567         bridge-multicast-flood off
 568
 569
 570 auto vmbr2
 571 iface vmbr2 inet static
 572         address 10.0.2.254
 573         netmask 255.255.255.0
 574         hwaddress 44:39:39:FF:40:94
 575         bridge_ports vxlan2
 576         bridge_stp off
 577         bridge_fd 0
 578         ip-forward on
 579         ip6-forward on
 580         arp-accept on
 581
 582 auto vxlan3
 583 iface vxlan3 inet manual
 584         vxlan-id 3
 585         vxlan-local-tunnelip 192.168.0.1
 586         bridge-learning off
 587         bridge-arp-nd-suppress on
 588         bridge-unicast-flood off
 589         bridge-multicast-flood off
 590
 591
 592 auto vmbr3
 593 iface vmbr3 inet static
 594         address 10.0.3.254
 595         netmask 255.255.255.0
 596         hwaddress 44:39:39:FF:40:94
 597         bridge_ports vxlan3
 598         bridge_stp off
 599         bridge_fd 0
 600         ip-forward on
 601         ip6-forward on
 602         arp-accept on
 603 ----
 604
 605
 606 frr.conf
 607
 608 ----
 609 router bgp 1234
 610  bgp router-id 192.168.0.1
 611  no bgp default ipv4-unicast
 612  coalesce-time 1000
 613  neighbor 192.168.0.2 remote-as 1234
 614  neighbor 192.168.0.3 remote-as 1234
 615  !
 616  address-family l2vpn evpn
 617   neighbor 192.168.0.2 activate
 618   neighbor 192.168.0.3 activate
 619   advertise-all-vni
 620  exit-address-family
 621 !
 622 line vty
 623 !
 624 ----
 625
 626
 627 * node2
 628
 629 ----
 630 auto eno1
 631 iface eno1 inet manual
 632         mtu 1550
 633
 634 auto vmbr0
 635 iface vmbr0 inet static
 636          address 192.168.0.2
 637          netmask  255.255.255.0
 638          bridge_ports eno1
 639          bridge_stp off
 640          bridge_fd 0
 641
 642 auto vxlan2
 643 iface vxlan2 inet manual
 644         vxlan-id 2
 645         vxlan-local-tunnelip 192.168.0.2
 646         bridge-learning off
 647         bridge-arp-nd-suppress on
 648         bridge-unicast-flood off
 649         bridge-multicast-flood off
 650
 651
 652 auto vmbr2
 653 iface vmbr2 inet static
 654         address 10.0.2.254
 655         netmask 255.255.255.0
 656         hwaddress 44:39:39:FF:40:94
 657         bridge_ports vxlan2
 658         bridge_stp off
 659         bridge_fd 0
 660         ip-forward on
 661         ip6-forward on
 662         arp-accept on
 663
 664
 665 auto vxlan3
 666 iface vxlan3 inet manual
 667         vxlan-id 3
 668         vxlan-local-tunnelip 192.168.0.2
 669         bridge-learning off
 670         bridge-arp-nd-suppress on
 671         bridge-unicast-flood off
 672         bridge-multicast-flood off
 673
 674
 675 auto vmbr3
 676 iface vmbr3 inet static
 677         address 10.0.3.254
 678         netmask 255.255.255.0
 679         hwaddress 44:39:39:FF:40:94
 680         bridge_ports vxlan3
 681         bridge_stp off
 682         bridge_fd 0
 683         ip-forward on
 684         ip6-forward on
 685         arp-accept on
 686 ----
 687
 688
 689 frr.conf
 690
 691 ----
 692 router bgp 1234
 693  bgp router-id 192.168.0.2
 694  no bgp default ipv4-unicast
 695  coalesce-time 1000
 696  neighbor 192.168.0.1 remote-as 1234
 697  neighbor 192.168.0.3 remote-as 1234
 698  !
 699  address-family l2vpn evpn
 700   neighbor 192.168.0.1 activate
 701   neighbor 192.168.0.3 activate
 702   advertise-all-vni
 703  exit-address-family
 704 !
 705 line vty
 706 !
 707 ----
 708
 709
 710 * node3
 711
 712 ----
 713 auto eno1
 714 iface eno1 inet manual
 715         mtu 1550
 716
 717 auto vmbr0
 718 iface vmbr0 inet static
 719          address 192.168.0.3
 720          netmask  255.255.255.0
 721          bridge_ports eno1
 722          bridge_stp off
 723          bridge_fd 0
 724
 725 auto vxlan2
 726 iface vxlan2 inet manual
 727         vxlan-id 2
 728         vxlan-local-tunnelip 192.168.0.3
 729         bridge-learning off
 730         bridge-arp-nd-suppress on
 731         bridge-unicast-flood off
 732         bridge-multicast-flood off
 733
 734
 735 auto vmbr2
 736 iface vmbr2 inet static
 737         address 10.0.2.254
 738         netmask 255.255.255.0
 739         hwaddress 44:39:39:FF:40:94
 740         bridge_ports vxlan2
 741         bridge_stp off
 742         bridge_fd 0
 743         ip-forward on
 744         ip6-forward on
 745         arp-accept on
 746
 747 auto vxlan3
 748 iface vxlan3 inet manual
 749         vxlan-id 3
 750         vxlan-local-tunnelip 192.168.0.3
 751         bridge-learning off
 752         bridge-arp-nd-suppress on
 753         bridge-unicast-flood off
 754         bridge-multicast-flood off
 755
 756 auto vmbr3
 757 iface vmbr3 inet static
 758         address 10.0.3.254
 759         netmask 255.255.255.0
 760         hwaddress 44:39:39:FF:40:94
 761         bridge_ports vxlan3
 762         bridge_stp off
 763         bridge_fd 0
 764         ip-forward on
 765         ip6-forward on
 766         arp-accept on
 767 ----
 768
 769
 770 frr.conf
 771
 772 ----
 773 router bgp 1234
 774  bgp router-id 192.168.0.3
 775  no bgp default ipv4-unicast
 776  coalesce-time 1000
 777  neighbor 192.168.0.1 remote-as 1234
 778  neighbor 192.168.0.2 remote-as 1234
 779  !
 780  address-family l2vpn evpn
 781   neighbor 192.168.0.1 activate
 782   neighbor 192.168.0.2 activate
 783   advertise-all-vni
 784  exit-address-family
 785 !
 786 line vty
 787 !
 788 ----
 789
 790
 791 symmetric model
 792 ^^^^^^^^^^^^^^^
 793
 794 With this model, you don't need to have all vxlan on all nodes.
 795 This model will also be needed to route traffic to an external router.
 796
 797 The symmetric model routes and bridges on both the ingress and the egress leafs.
 798 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 799 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 800 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 801 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 802
 803 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 804
 805 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 806
 807 * node1
 808
 809 ----
 810 auto vrf1
 811 iface vrf1
 812     vrf-table auto
 813
 814 auto eno1
 815 iface eno1 inet manual
 816         mtu 1550
 817
 818 auto vmbr0
 819 iface vmbr0 inet static
 820          address 192.168.0.1
 821          netmask  255.255.255.0
 822          bridge_ports eno1
 823          bridge_stp off
 824          bridge_fd 0
 825
 826 auto vxlan2
 827 iface vxlan2 inet manual
 828         vxlan-id 2
 829         vxlan-local-tunnelip 192.168.0.1
 830         bridge-learning off
 831         bridge-arp-nd-suppress on
 832         bridge-unicast-flood off
 833         bridge-multicast-flood off
 834
 835 auto vmbr2
 836 iface vmbr2 inet static
 837         bridge_ports vxlan2
 838         bridge_stp off
 839         bridge_fd 0
 840         address 10.0.2.254
 841         netmask 255.255.255.0
 842         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 843         vrf vrf1
 844         ip-forward on
 845         ip6-forward on
 846         arp-accept on
 847
 848 auto vxlan3
 849 iface vxlan3 inet manual
 850         vxlan-id 3
 851         vxlan-local-tunnelip 192.168.0.1
 852         bridge-learning off
 853         bridge-arp-nd-suppress on
 854         bridge-unicast-flood off
 855         bridge-multicast-flood off
 856
 857 auto vmbr3
 858 iface vmbr3 inet static
 859         bridge_ports vxlan3
 860         bridge_stp off
 861         bridge_fd 0
 862         address 10.0.3.254
 863         netmask 255.255.255.0
 864         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 865         vrf vrf1
 866         ip-forward on
 867         ip6-forward on
 868         arp-accept on
 869
 870 #interconnect vxlan-vfr l3vni
 871 auto vxlan4000
 872 iface vxlan4000 inet manual
 873         vxlan-id 4000
 874         vxlan-local-tunnelip 192.168.0.1
 875         bridge-learning off
 876         bridge-arp-nd-suppress on
 877         bridge-unicast-flood off
 878         bridge-multicast-flood off
 879
 880
 881 auto vmbr4000
 882 iface vmbr4000 inet manual
 883         bridge_ports vxlan4000
 884         bridge_stp off
 885         bridge_fd 0
 886         vrf vrf1
 887 ----
 888
 889 frr.conf
 890
 891 ----
 892 vrf vrf1
 893  vni 4000
 894  exit-vrf
 895 !
 896 router bgp 1234
 897  bgp router-id 192.168.0.1
 898  no bgp default ipv4-unicast
 899  coalesce-time 1000
 900  neighbor 192.168.0.2 remote-as 1234
 901  neighbor 192.168.0.3 remote-as 1234
 902  !
 903  address-family l2vpn evpn
 904   neighbor 192.168.0.2 activate
 905   neighbor 192.168.0.3 activate
 906   advertise-all-vni
 907  exit-address-family
 908 !
 909 line vty
 910 !
 911 ----
 912
 913
 914 * node2
 915
 916 ----
 917 auto vrf1
 918 iface vrf1
 919     vrf-table auto
 920
 921 auto eno1
 922 iface eno1 inet manual
 923         mtu 1550
 924
 925 auto vmbr0
 926 iface vmbr0 inet static
 927          address 192.168.0.2
 928          netmask  255.255.255.0
 929          bridge_ports eno1
 930          bridge_stp off
 931          bridge_fd 0
 932
 933 auto vxlan2
 934 iface vxlan2 inet manual
 935         vxlan-id 2
 936         vxlan-local-tunnelip 192.168.0.2
 937         bridge-learning off
 938         bridge-arp-nd-suppress on
 939         bridge-unicast-flood off
 940         bridge-multicast-flood off
 941
 942 auto vmbr2
 943 iface vmbr2 inet static
 944         bridge_ports vxlan2
 945         bridge_stp off
 946         bridge_fd 0
 947         address 10.0.2.254
 948         netmask 255.255.255.0
 949         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 950         vrf vrf1
 951         ip-forward on
 952         ip6-forward on
 953         arp-accept on
 954
 955 auto vxlan3
 956 iface vxlan3 inet manual
 957         vxlan-id 3
 958         vxlan-local-tunnelip 192.168.0.2
 959         bridge-learning off
 960         bridge-arp-nd-suppress on
 961         bridge-unicast-flood off
 962         bridge-multicast-flood off
 963
 964 auto vmbr3
 965 iface vmbr3 inet static
 966         bridge_ports vxlan3
 967         bridge_stp off
 968         bridge_fd 0
 969         address 10.0.3.254
 970         netmask 255.255.255.0
 971         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 972         vrf vrf1
 973         ip-forward on
 974         ip6-forward on
 975         arp-accept on
 976
 977 #interconnect vxlan-vfr l3vni
 978 auto vxlan4000
 979 iface vxlan4000 inet manual
 980         vxlan-id 4000
 981         vxlan-local-tunnelip 192.168.0.2
 982         bridge-learning off
 983         bridge-arp-nd-suppress on
 984         bridge-unicast-flood off
 985         bridge-multicast-flood off
 986
 987
 988 auto vmbr4000
 989 iface vmbr4000 inet manual
 990         bridge_ports vxlan4000
 991         bridge_stp off
 992         bridge_fd 0
 993         vrf vrf1
 994 ----
 995
 996
 997 frr.conf
 998
 999 ----
1000 vrf vrf1
1001  vni 4000
1002  exit-vrf
1003 !
1004 router bgp 1234
1005  bgp router-id 192.168.0.2
1006  no bgp default ipv4-unicast
1007  coalesce-time 1000
1008  neighbor 192.168.0.1 remote-as 1234
1009  neighbor 192.168.0.3 remote-as 1234
1010  !
1011  address-family l2vpn evpn
1012   neighbor 192.168.0.1 activate
1013   neighbor 192.168.0.3 activate
1014   advertise-all-vni
1015  exit-address-family
1016 !
1017 line vty
1018 !
1019 ----
1020
1021
1022 * node3
1023
1024 ----
1025 auto vrf1
1026 iface vrf1
1027     vrf-table auto
1028
1029 auto eno1
1030 iface eno1 inet manual
1031         mtu 1550
1032
1033 auto vmbr0
1034 iface vmbr0 inet static
1035          address 192.168.0.3
1036          netmask  255.255.255.0
1037          bridge_ports eno1
1038          bridge_stp off
1039          bridge_fd 0
1040
1041 auto vxlan2
1042 iface vxlan2 inet manual
1043         vxlan-id 2
1044         vxlan-local-tunnelip 192.168.0.3
1045         bridge-learning off
1046         bridge-arp-nd-suppress on
1047         bridge-unicast-flood off
1048         bridge-multicast-flood off
1049
1050 auto vmbr2
1051 iface vmbr2 inet static
1052         bridge_ports vxlan2
1053         bridge_stp off
1054         bridge_fd 0
1055         address 10.0.2.254
1056         netmask 255.255.255.0
1057         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1058         vrf vrf1
1059         ip-forward on
1060         ip6-forward on
1061         arp-accept on
1062
1063 auto vxlan3
1064 iface vxlan3 inet manual
1065         vxlan-id 3
1066         vxlan-local-tunnelip 192.168.0.3
1067         bridge-learning off
1068         bridge-arp-nd-suppress on
1069         bridge-unicast-flood off
1070         bridge-multicast-flood off
1071
1072 auto vmbr3
1073 iface vmbr3 inet static
1074         bridge_ports vxlan3
1075         bridge_stp off
1076         bridge_fd 0
1077         address 10.0.3.254
1078         netmask 255.255.255.0
1079         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1080         vrf vrf1
1081         ip-forward on
1082         ip6-forward on
1083         arp-accept on
1084
1085 #interconnect vxlan-vfr l3vni
1086 auto vxlan4000
1087 iface vxlan4000 inet manual
1088         vxlan-id 4000
1089         vxlan-local-tunnelip 192.168.0.3
1090         bridge-learning off
1091         bridge-arp-nd-suppress on
1092         bridge-unicast-flood off
1093         bridge-multicast-flood off
1094
1095
1096 auto vmbr4000
1097 iface vmbr4000 inet manual
1098         bridge_ports vxlan4000
1099         bridge_stp off
1100         bridge_fd 0
1101         vrf vrf1
1102 ----
1103
1104
1105 frr.conf
1106
1107 ----
1108 vrf vrf1
1109  vni 4000
1110  exit-vrf
1111 !
1112 router bgp 1234
1113  bgp router-id 192.168.0.3
1114  no bgp default ipv4-unicast
1115  coalesce-time 1000
1116  neighbor 192.168.0.1 remote-as 1234
1117  neighbor 192.168.0.2 remote-as 1234
1118  !
1119  address-family l2vpn evpn
1120   neighbor 192.168.0.1 activate
1121   neighbor 192.168.0.2 activate
1122   advertise-all-vni
1123  exit-address-family
1124 !
1125 line vty
1126 !
1127 ----
1128
1129 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1130 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1131 Routing to outside need the symmetric model.
1132
1133 1 gateway node
1134 ^^^^^^^^^^^^^^
1135 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1136 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1137
1138
1139 *node1
1140
1141 ----
1142 auto vrf1
1143 iface vrf1
1144     vrf-table auto
1145
1146 auto eno1
1147 iface eno1 inet manual
1148         mtu 1550
1149
1150 auto vmbr0
1151 iface vmbr0 inet static
1152          address 192.168.0.1
1153          netmask  255.255.255.0
1154          gateway 192.168.0.254
1155          bridge_ports eno1
1156          bridge_stp off
1157          bridge_fd 0
1158
1159 auto vxlan2
1160 iface vxlan2 inet manual
1161         vxlan-id 2
1162         vxlan-local-tunnelip 192.168.0.1
1163         bridge-learning off
1164         bridge-arp-nd-suppress on
1165         bridge-unicast-flood off
1166         bridge-multicast-flood off
1167
1168 auto vmbr2
1169 iface vmbr2 inet static
1170         bridge_ports vxlan2
1171         bridge_stp off
1172         bridge_fd 0
1173         address 10.0.2.254
1174         netmask 255.255.255.0
1175         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1176         vrf vrf1
1177         ip-forward on
1178         ip6-forward on
1179         arp-accept on
1180
1181 auto vxlan3
1182 iface vxlan3 inet manual
1183         vxlan-id 3
1184         vxlan-local-tunnelip 192.168.0.1
1185         bridge-learning off
1186         bridge-arp-nd-suppress on
1187         bridge-unicast-flood off
1188         bridge-multicast-flood off
1189
1190 auto vmbr3
1191 iface vmbr3 inet static
1192         bridge_ports vxlan3
1193         bridge_stp off
1194         bridge_fd 0
1195         address 10.0.3.254
1196         netmask 255.255.255.0
1197         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1198         vrf vrf1
1199         ip-forward on
1200         ip6-forward on
1201         arp-accept on
1202
1203 #interconnect vxlan-vfr l3vni
1204 auto vxlan4000
1205 iface vxlan4000 inet manual
1206         vxlan-id 4000
1207         vxlan-local-tunnelip 192.168.0.1
1208         bridge-learning off
1209         bridge-arp-nd-suppress on
1210         bridge-unicast-flood off
1211         bridge-multicast-flood off
1212
1213 auto vmbr4000
1214 iface vmbr4000 inet manual
1215         bridge_ports vxlan4000
1216         bridge_stp off
1217         bridge_fd 0
1218         vrf vrf1
1219 ----
1220
1221
1222 frr.conf
1223
1224 ----
1225 vrf vrf1
1226  vni 4000
1227  exit-vrf
1228 !
1229 router bgp 1234
1230  bgp router-id 192.168.0.1
1231  no bgp default ipv4-unicast
1232  coalesce-time 1000
1233  neighbor 192.168.0.2 remote-as 1234
1234  neighbor 192.168.0.3 remote-as 1234
1235  !
1236  address-family ipv4 unicast
1237   import vrf vrf1
1238  exit-address-family
1239  !
1240  address-family l2vpn evpn
1241   neighbor 192.168.0.2 activate
1242   neighbor 192.168.0.3 activate
1243   advertise-all-vni
1244  exit-address-family
1245 !
1246 router bgp 1234 vrf vrf1
1247 !
1248  address-family l2vpn evpn
1249   default-originate ipv4
1250  exit-address-family
1251 !
1252 line vty
1253 !
1254 ----
1255
1256
1257 * node2
1258
1259 ----
1260 auto vrf1
1261 iface vrf1
1262     vrf-table auto
1263
1264 auto eno1
1265 iface eno1 inet manual
1266         mtu 1550
1267
1268 auto vmbr0
1269 iface vmbr0 inet static
1270          address 192.168.0.2
1271          netmask  255.255.255.0
1272          bridge_ports eno1
1273          bridge_stp off
1274          bridge_fd 0
1275
1276 auto vxlan2
1277 iface vxlan2 inet manual
1278         vxlan-id 2
1279         vxlan-local-tunnelip 192.168.0.2
1280         bridge-learning off
1281         bridge-arp-nd-suppress on
1282         bridge-unicast-flood off
1283         bridge-multicast-flood off
1284
1285 auto vmbr2
1286 iface vmbr2 inet static
1287         bridge_ports vxlan2
1288         bridge_stp off
1289         bridge_fd 0
1290         address 10.0.2.254
1291         netmask 255.255.255.0
1292         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1293         vrf vrf1
1294         ip-forward on
1295         ip6-forward on
1296         arp-accept on
1297
1298 auto vxlan3
1299 iface vxlan3 inet manual
1300         vxlan-id 3
1301         vxlan-local-tunnelip 192.168.0.2
1302         bridge-learning off
1303         bridge-arp-nd-suppress on
1304         bridge-unicast-flood off
1305         bridge-multicast-flood off
1306
1307 auto vmbr3
1308 iface vmbr3 inet static
1309         bridge_ports vxlan3
1310         bridge_stp off
1311         bridge_fd 0
1312         address 10.0.3.254
1313         netmask 255.255.255.0
1314         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1315         vrf vrf1
1316         ip-forward on
1317         ip6-forward on
1318         arp-accept on
1319
1320 #interconnect vxlan-vfr l3vni
1321 auto vxlan4000
1322 iface vxlan4000 inet manual
1323         vxlan-id 4000
1324         vxlan-local-tunnelip 192.168.0.2
1325         bridge-learning off
1326         bridge-arp-nd-suppress on
1327         bridge-unicast-flood off
1328         bridge-multicast-flood off
1329
1330
1331 auto vmbr4000
1332 iface vmbr4000 inet manual
1333         bridge_ports vxlan4000
1334         bridge_stp off
1335         bridge_fd 0
1336         vrf vrf1
1337 ----
1338
1339
1340 frr.conf
1341
1342 ----
1343 vrf vrf1
1344  vni 4000
1345  exit-vrf
1346 !
1347 router bgp 1234
1348  bgp router-id 192.168.0.2
1349  no bgp default ipv4-unicast
1350  coalesce-time 1000
1351  neighbor 192.168.0.1 remote-as 1234
1352  neighbor 192.168.0.3 remote-as 1234
1353  !
1354  address-family l2vpn evpn
1355   neighbor 192.168.0.1 activate
1356   neighbor 192.168.0.3 activate
1357   advertise-all-vni
1358  exit-address-family
1359 !
1360 line vty
1361 !
1362 ----
1363
1364
1365 * node3
1366
1367 ----
1368 auto vrf1
1369 iface vrf1
1370     vrf-table auto
1371
1372 auto eno1
1373 iface eno1 inet manual
1374         mtu 1550
1375
1376 auto vmbr0
1377 iface vmbr0 inet static
1378          address 192.168.0.3
1379          netmask  255.255.255.0
1380          bridge_ports eno1
1381          bridge_stp off
1382          bridge_fd 0
1383
1384 auto vxlan2
1385 iface vxlan2 inet manual
1386         vxlan-id 2
1387         vxlan-local-tunnelip 192.168.0.3
1388         bridge-learning off
1389         bridge-arp-nd-suppress on
1390         bridge-unicast-flood off
1391         bridge-multicast-flood off
1392
1393 auto vmbr2
1394 iface vmbr2 inet static
1395         bridge_ports vxlan2
1396         bridge_stp off
1397         bridge_fd 0
1398         address 10.0.2.254
1399         netmask 255.255.255.0
1400         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1401         vrf vrf1
1402         ip-forward on
1403         ip6-forward on
1404         arp-accept on
1405
1406 auto vxlan3
1407 iface vxlan3 inet manual
1408         vxlan-id 3
1409         vxlan-local-tunnelip 192.168.0.3
1410         bridge-learning off
1411         bridge-arp-nd-suppress on
1412         bridge-unicast-flood off
1413         bridge-multicast-flood off
1414
1415 auto vmbr3
1416 iface vmbr3 inet static
1417         bridge_ports vxlan3
1418         bridge_stp off
1419         bridge_fd 0
1420         address 10.0.3.254
1421         netmask 255.255.255.0
1422         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1423         vrf vrf1
1424         ip-forward on
1425         ip6-forward on
1426         arp-accept on
1427
1428 #interconnect vxlan-vfr l3vni
1429 auto vxlan4000
1430 iface vxlan4000 inet manual
1431         vxlan-id 4000
1432         vxlan-local-tunnelip 192.168.0.3
1433         bridge-learning off
1434         bridge-arp-nd-suppress on
1435         bridge-unicast-flood off
1436         bridge-multicast-flood off
1437
1438
1439 auto vmbr4000
1440 iface vmbr4000 inet manual
1441         bridge_ports vxlan4000
1442         bridge_stp off
1443         bridge_fd 0
1444         vrf vrf1
1445 ----
1446
1447
1448 frr.conf
1449
1450 ----
1451 vrf vrf1
1452  vni 4000
1453  exit-vrf
1454 !
1455 router bgp 1234
1456  bgp router-id 192.168.0.3
1457  no bgp default ipv4-unicast
1458  coalesce-time 1000
1459  neighbor 192.168.0.1 remote-as 1234
1460  neighbor 192.168.0.2 remote-as 1234
1461  !
1462  address-family l2vpn evpn
1463   neighbor 192.168.0.1 activate
1464   neighbor 192.168.0.2 activate
1465   advertise-all-vni
1466  exit-address-family
1467 !
1468 line vty
1469 !
1470 ----
1471
1472 multiple gateway nodes
1473 ^^^^^^^^^^^^^^^^^^^^^^
1474 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1475 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1476 and announce this default gw in the vrf (default originate)
1477 The external router have ecmp routes to all proxmox nodes.(balancing).
1478 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1479 vxlan the packet to final destination.
1480
1481 *node1
1482
1483 ----
1484 auto vrf1
1485 iface vrf1
1486     vrf-table auto
1487
1488 auto eno1
1489 iface eno1 inet manual
1490         mtu 1550
1491
1492 auto vmbr0
1493 iface vmbr0 inet static
1494          address 192.168.0.1
1495          netmask  255.255.255.0
1496          gateway 192.168.0.254
1497          bridge_ports eno1
1498          bridge_stp off
1499          bridge_fd 0
1500
1501 auto vxlan2
1502 iface vxlan2 inet manual
1503         vxlan-id 2
1504         vxlan-local-tunnelip 192.168.0.1
1505         bridge-learning off
1506         bridge-arp-nd-suppress on
1507         bridge-unicast-flood off
1508         bridge-multicast-flood off
1509
1510 auto vmbr2
1511 iface vmbr2 inet static
1512         bridge_ports vxlan2
1513         bridge_stp off
1514         bridge_fd 0
1515         address 10.0.2.254
1516         netmask 255.255.255.0
1517         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1518         vrf vrf1
1519         ip-forward on
1520         ip6-forward on
1521         arp-accept on
1522
1523 auto vxlan3
1524 iface vxlan3 inet manual
1525         vxlan-id 3
1526         vxlan-local-tunnelip 192.168.0.1
1527         bridge-learning off
1528         bridge-arp-nd-suppress on
1529         bridge-unicast-flood off
1530         bridge-multicast-flood off
1531
1532 auto vmbr3
1533 iface vmbr3 inet static
1534         bridge_ports vxlan3
1535         bridge_stp off
1536         bridge_fd 0
1537         address 10.0.3.254
1538         netmask 255.255.255.0
1539         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1540         vrf vrf1
1541         ip-forward on
1542         ip6-forward on
1543         arp-accept on
1544
1545 #interconnect vxlan-vfr l3vni
1546 auto vxlan4000
1547 iface vxlan4000 inet manual
1548         vxlan-id 4000
1549         vxlan-local-tunnelip 192.168.0.1
1550         bridge-learning off
1551         bridge-arp-nd-suppress on
1552         bridge-unicast-flood off
1553         bridge-multicast-flood off
1554
1555 auto vmbr4000
1556 iface vmbr4000 inet manual
1557         bridge_ports vxlan4000
1558         bridge_stp off
1559         bridge_fd 0
1560         vrf vrf1
1561 ----
1562
1563
1564 frr.conf
1565
1566 ----
1567 vrf vrf1
1568  vni 4000
1569  exit-vrf
1570 !
1571 router bgp 1234
1572  bgp router-id 192.168.0.1
1573  no bgp default ipv4-unicast
1574  coalesce-time 1000
1575  neighbor 192.168.0.2 remote-as 1234
1576  neighbor 192.168.0.3 remote-as 1234
1577  !
1578  address-family ipv4 unicast
1579   import vrf vrf1
1580  exit-address-family
1581  !
1582  address-family l2vpn evpn
1583   neighbor 192.168.0.2 activate
1584   neighbor 192.168.0.3 activate
1585   advertise-all-vni
1586  exit-address-family
1587 !
1588 router bgp 1234 vrf vrf1
1589 !
1590  address-family l2vpn evpn
1591   default-originate ipv4
1592  exit-address-family
1593 !
1594 line vty
1595 !
1596 ----
1597
1598
1599 * node2
1600
1601 ----
1602 auto vrf1
1603 iface vrf1
1604     vrf-table auto
1605
1606 auto eno1
1607 iface eno1 inet manual
1608         mtu 1550
1609
1610 auto vmbr0
1611 iface vmbr0 inet static
1612          address 192.168.0.2
1613          netmask  255.255.255.0
1614          gateway 192.168.0.254
1615          bridge_ports eno1
1616          bridge_stp off
1617          bridge_fd 0
1618
1619 auto vxlan2
1620 iface vxlan2 inet manual
1621         vxlan-id 2
1622         vxlan-local-tunnelip 192.168.0.2
1623         bridge-learning off
1624         bridge-arp-nd-suppress on
1625         bridge-unicast-flood off
1626         bridge-multicast-flood off
1627
1628 auto vmbr2
1629 iface vmbr2 inet static
1630         bridge_ports vxlan2
1631         bridge_stp off
1632         bridge_fd 0
1633         address 10.0.2.254
1634         netmask 255.255.255.0
1635         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1636         vrf vrf1
1637         ip-forward on
1638         ip6-forward on
1639         arp-accept on
1640
1641 auto vxlan3
1642 iface vxlan3 inet manual
1643         vxlan-id 3
1644         vxlan-local-tunnelip 192.168.0.2
1645         bridge-learning off
1646         bridge-arp-nd-suppress on
1647         bridge-unicast-flood off
1648         bridge-multicast-flood off
1649
1650 auto vmbr3
1651 iface vmbr3 inet static
1652         bridge_ports vxlan3
1653         bridge_stp off
1654         bridge_fd 0
1655         address 10.0.3.254
1656         netmask 255.255.255.0
1657         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1658         vrf vrf1
1659         ip-forward on
1660         ip6-forward on
1661         arp-accept on
1662
1663 #interconnect vxlan-vfr l3vni
1664 auto vxlan4000
1665 iface vxlan4000 inet manual
1666         vxlan-id 4000
1667         vxlan-local-tunnelip 192.168.0.2
1668         bridge-learning off
1669         bridge-arp-nd-suppress on
1670         bridge-unicast-flood off
1671         bridge-multicast-flood off
1672
1673
1674 auto vmbr4000
1675 iface vmbr4000 inet manual
1676         bridge_ports vxlan4000
1677         bridge_stp off
1678         bridge_fd 0
1679         vrf vrf1
1680 ----
1681
1682
1683 frr.conf
1684
1685 ----
1686 vrf vrf1
1687  vni 4000
1688  exit-vrf
1689 !
1690 router bgp 1234
1691  bgp router-id 192.168.0.2
1692  no bgp default ipv4-unicast
1693  coalesce-time 1000
1694  neighbor 192.168.0.1 remote-as 1234
1695  neighbor 192.168.0.3 remote-as 1234
1696  !
1697  address-family ipv4 unicast
1698   import vrf vrf1
1699  exit-address-family
1700  !
1701  address-family l2vpn evpn
1702   neighbor 192.168.0.1 activate
1703   neighbor 192.168.0.3 activate
1704   advertise-all-vni
1705  exit-address-family
1706 !
1707  address-family l2vpn evpn
1708   default-originate ipv4
1709  exit-address-family
1710 !
1711 line vty
1712 !
1713 ----
1714
1715
1716 * node3
1717
1718 ----
1719 auto vrf1
1720 iface vrf1
1721     vrf-table auto
1722
1723 auto eno1
1724 iface eno1 inet manual
1725         mtu 1550
1726
1727 auto vmbr0
1728 iface vmbr0 inet static
1729          address 192.168.0.3
1730          netmask  255.255.255.0
1731          gateway 192.168.0.254
1732          bridge_ports eno1
1733          bridge_stp off
1734          bridge_fd 0
1735
1736 auto vxlan2
1737 iface vxlan2 inet manual
1738         vxlan-id 2
1739         vxlan-local-tunnelip 192.168.0.3
1740         bridge-learning off
1741         bridge-arp-nd-suppress on
1742         bridge-unicast-flood off
1743         bridge-multicast-flood off
1744
1745 auto vmbr2
1746 iface vmbr2 inet static
1747         bridge_ports vxlan2
1748         bridge_stp off
1749         bridge_fd 0
1750         address 10.0.2.254
1751         netmask 255.255.255.0
1752         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1753         vrf vrf1
1754         ip-forward on
1755         ip6-forward on
1756         arp-accept on
1757
1758 auto vxlan3
1759 iface vxlan3 inet manual
1760         vxlan-id 3
1761         vxlan-local-tunnelip 192.168.0.3
1762         bridge-learning off
1763         bridge-arp-nd-suppress on
1764         bridge-unicast-flood off
1765         bridge-multicast-flood off
1766
1767 auto vmbr3
1768 iface vmbr3 inet static
1769         bridge_ports vxlan3
1770         bridge_stp off
1771         bridge_fd 0
1772         address 10.0.3.254
1773         netmask 255.255.255.0
1774         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1775         vrf vrf1
1776         ip-forward on
1777         ip6-forward on
1778         arp-accept on
1779
1780 #interconnect vxlan-vfr l3vni
1781 auto vxlan4000
1782 iface vxlan4000 inet manual
1783         vxlan-id 4000
1784         vxlan-local-tunnelip 192.168.0.3
1785         bridge-learning off
1786         bridge-arp-nd-suppress on
1787         bridge-unicast-flood off
1788         bridge-multicast-flood off
1789
1790
1791 auto vmbr4000
1792 iface vmbr4000 inet manual
1793         bridge_ports vxlan4000
1794         bridge_stp off
1795         bridge_fd 0
1796         vrf vrf1
1797 ----
1798
1799
1800 frr.conf
1801
1802 ----
1803 vrf vrf1
1804  vni 4000
1805  exit-vrf
1806 !
1807 router bgp 1234
1808  bgp router-id 192.168.0.3
1809  no bgp default ipv4-unicast
1810  coalesce-time 1000
1811  neighbor 192.168.0.1 remote-as 1234
1812  neighbor 192.168.0.2 remote-as 1234
1813  !
1814  address-family ipv4 unicast
1815   import vrf vrf1
1816  exit-address-family
1817  !
1818  address-family l2vpn evpn
1819   neighbor 192.168.0.1 activate
1820   neighbor 192.168.0.2 activate
1821   advertise-all-vni
1822  exit-address-family
1823 !
1824 router bgp 1234 vrf vrf1
1825 !
1826  address-family l2vpn evpn
1827   default-originate ipv4
1828  exit-address-family
1829 !
1830 line vty
1831 !
1832 ----
1833
1834 Note
1835 ^^^^
1836
1837 If your external router doesn't support 'ECMP static routes' to reach multiple
1838 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1839 Virtual Router Redundancy Protocol (VRRP).
1840
1841 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1842 Node1 is the primary with failover to node2 in case of outage.
1843
1844 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1845 #TODO : It should be possible to do it with frr directly with last version.
1846
1847 * node1
1848
1849 ----
1850 auto vmbr0
1851 iface vmbr0 inet static
1852          address 192.168.0.1
1853          netmask  255.255.255.0
1854          gateway 192.168.0.254
1855          bridge_ports eno1
1856          bridge_stp off
1857          bridge_fd 0
1858          vrrp-id 1
1859          vrrp-priority 1
1860          vrrp-virtual-ip 192.168.0.10
1861 ----
1862
1863 * node2
1864
1865 ----
1866 auto vmbr0
1867 iface vmbr0 inet static
1868          address 192.168.0.2
1869          netmask  255.255.255.0
1870          gateway 192.168.0.254
1871          bridge_ports eno1
1872          bridge_stp off
1873          bridge_fd 0
1874          vrrp-id 1
1875          vrrp-priority 2
1876          vrrp-virtual-ip 192.168.0.10
1877 ----
1878
1879
1880 Route Reflectors
1881 ^^^^^^^^^^^^^^^^
1882 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
1883 to avoid that all node peers with each others nodes.
1884 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
1885 single point of failure, a minimum of two servers acting as an RR is highly
1886 recommended for redundancy.
1887
1888 Below is an example of configuration with 'frr', with `rrserver1
1889 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
1890
1891 rrserver1
1892 ----
1893 router bgp 1234
1894   bgp router-id 192.168.0.200
1895   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
1896   bgp log-neighbor-changes
1897   no bgp default ipv4-unicast
1898   neighbor fabric peer-group
1899   neighbor fabric remote-as 1234
1900   neighbor fabric capability extended-nexthop
1901   neighbor fabric update-source 192.168.0.200
1902   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
1903   !
1904   address-family l2vpn evpn
1905    neighbor fabric activate
1906    neighbor fabric route-reflector-client
1907    neighbor fabric allowas-in
1908   exit-address-family
1909   !
1910   exit
1911 !
1912 ---
1913
1914 rrserver2
1915 ----
1916 router bgp 1234
1917   bgp router-id 192.168.0.201
1918   bgp cluster-id 1.1.1.1
1919   bgp log-neighbor-changes
1920   no bgp default ipv4-unicast
1921   neighbor fabric peer-group
1922   neighbor fabric remote-as 1234
1923   neighbor fabric capability extended-nexthop
1924   neighbor fabric update-source 192.168.0.201
1925   bgp listen range 192.168.0.0/24 peer-group fabric
1926   !
1927   address-family l2vpn evpn
1928    neighbor fabric activate
1929    neighbor fabric route-reflector-client
1930    neighbor fabric allowas-in
1931   exit-address-family
1932   !
1933   exit
1934 !
1935 ---
1936
1937 proxmoxnode(s)
1938 ----
1939 router bgp 1234
1940  bgp router-id 192.168.0.x
1941  no bgp default ipv4-unicast
1942  coalesce-time 1000
1943  neighbor 192.168.0.200 remote-as 1234
1944  neighbor 192.168.0.201 remote-as 1234
1945  !
1946  address-family ipv4 unicast
1947   import vrf vrf1
1948  exit-address-family
1949  !
1950  address-family l2vpn evpn
1951   neighbor 192.168.0.200 activate
1952   neighbor 192.168.0.201 activate
1953   advertise-all-vni
1954  exit-address-family
1955 !
1956 ----
1957
1958 #TODO : Documentation with bgp upstream router.