vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  coalesce-time 1000
 371  neighbor 192.168.0.2 remote-as 1234
 372  neighbor 192.168.0.3 remote-as 1234
 373  !
 374  address-family l2vpn evpn
 375   neighbor 192.168.0.2 activate
 376   neighbor 192.168.0.3 activate
 377   advertise-all-vni
 378  exit-address-family
 379 !
 380 line vty
 381 !
 382 ----
 383
 384
 385 * node2
 386
 387 ----
 388 auto eno1
 389 iface eno1 inet manual
 390         mtu 1550
 391
 392 auto vmbr0
 393 iface vmbr0 inet static
 394         address 192.168.0.2
 395         netmask 255.255.255.0
 396         bridge_ports eno1
 397         bridge_stp off
 398         bridge_fd 0
 399
 400 auto vxlan2
 401 iface vxlan2 inet manual
 402         vxlan-id 2
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr2
 411 iface vmbr2 inet manual
 412         bridge_ports vxlan2
 413         bridge_stp off
 414         bridge_fd 0
 415
 416 auto vxlan3
 417 iface vxlan3 inet manual
 418         vxlan-id 3
 419         vxlan-local-tunnelip 192.168.0.2
 420         bridge-learning off
 421         bridge-arp-nd-suppress on
 422         bridge-unicast-flood off
 423         bridge-multicast-flood off
 424
 425
 426 auto vmbr3
 427 iface vmbr3 inet manual
 428         bridge_ports vxlan3
 429         bridge_stp off
 430         bridge_fd 0
 431 ----
 432
 433
 434 /etc/frr/frr.conf
 435
 436 ----
 437 router bgp 1234
 438  no bgp default ipv4-unicast
 439  coalesce-time 1000
 440  neighbor 192.168.0.1 remote-as 1234
 441  neighbor 192.168.0.3 remote-as 1234
 442  !
 443  address-family l2vpn evpn
 444   neighbor 192.168.0.1 activate
 445   neighbor 192.168.0.3 activate
 446   advertise-all-vni
 447  exit-address-family
 448 !
 449 line vty
 450 !
 451 ----
 452
 453
 454 * node3
 455
 456 ----
 457 auto eno1
 458 iface eno1 inet manual
 459         mtu 1550
 460
 461 auto vmbr0
 462 iface vmbr0 inet static
 463         address 192.168.0.2
 464         netmask 255.255.255.0
 465         bridge_ports eno1
 466         bridge_stp off
 467         bridge_fd 0
 468
 469 auto vxlan2
 470 iface vxlan2 inet manual
 471         vxlan-id 2
 472         vxlan-local-tunnelip 192.168.0.3
 473         bridge-learning off
 474         bridge-arp-nd-suppress on
 475         bridge-unicast-flood off
 476         bridge-multicast-flood off
 477
 478
 479 auto vmbr2
 480 iface vmbr2 inet manual
 481         bridge_ports vxlan2
 482         bridge_stp off
 483         bridge_fd 0
 484
 485 auto vxlan3
 486 iface vxlan3 inet manual
 487         vxlan-id 3
 488         vxlan-local-tunnelip 192.168.0.3
 489         bridge-learning off
 490         bridge-arp-nd-suppress on
 491         bridge-unicast-flood off
 492         bridge-multicast-flood off
 493
 494
 495 auto vmbr3
 496 iface vmbr3 inet manual
 497         bridge_ports vxlan3
 498         bridge_stp off
 499         bridge_fd 0
 500 ----
 501
 502
 503 /etc/frr/frr.conf
 504
 505
 506 ----
 507 router bgp 1234
 508  no bgp default ipv4-unicast
 509  coalesce-time 1000
 510  neighbor 192.168.0.1 remote-as 1234
 511  neighbor 192.168.0.2 remote-as 1234
 512  !
 513  address-family l2vpn evpn
 514   neighbor 192.168.0.1 activate
 515   neighbor 192.168.0.2 activate
 516   advertise-all-vni
 517  exit-address-family
 518 !
 519 line vty
 520 !
 521 ----
 522
 523 VXLAN layer3 routing with anycast gateway
 524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 525
 526 With this need, each vmbr bridge will be the gateway for the vm.
 527 Same vmbr on different node, will have same ip address and same mac address,
 528 to have working vm live migration and no network disruption.
 529
 530 VXLAN layer3 routing only work with FRR and non-aware bridge.
 531 (vlan aware bridge support is buggy currently).
 532
 533 asymmetric model
 534 ^^^^^^^^^^^^^^^^
 535
 536 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 537
 538 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 539 but only bridging on the egress.
 540 This results in bi-directional VXLAN traffic traveling on different VNIs
 541 in each direction (always the destination VNI) across the routed infrastructure.
 542
 543 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 544
 545 * node1
 546
 547 ----
 548 auto eno1
 549 iface eno1 inet manual
 550         mtu 1550
 551
 552 auto vmbr0
 553 iface vmbr0 inet static
 554          address 192.168.0.1
 555          netmask  255.255.255.0
 556          bridge_ports eno1
 557          bridge_stp off
 558          bridge_fd 0
 559
 560 auto vxlan2
 561 iface vxlan2 inet manual
 562         vxlan-id 2
 563         vxlan-local-tunnelip 192.168.0.1
 564         bridge-learning off
 565         bridge-arp-nd-suppress on
 566         bridge-unicast-flood off
 567         bridge-multicast-flood off
 568
 569
 570 auto vmbr2
 571 iface vmbr2 inet static
 572         address 10.0.2.254
 573         netmask 255.255.255.0
 574         hwaddress 44:39:39:FF:40:94
 575         bridge_ports vxlan2
 576         bridge_stp off
 577         bridge_fd 0
 578         ip-forward on
 579         ip6-forward on
 580         arp-accept on
 581
 582 auto vxlan3
 583 iface vxlan3 inet manual
 584         vxlan-id 3
 585         vxlan-local-tunnelip 192.168.0.1
 586         bridge-learning off
 587         bridge-arp-nd-suppress on
 588         bridge-unicast-flood off
 589         bridge-multicast-flood off
 590
 591
 592 auto vmbr3
 593 iface vmbr3 inet static
 594         address 10.0.3.254
 595         netmask 255.255.255.0
 596         hwaddress 44:39:39:FF:40:94
 597         bridge_ports vxlan3
 598         bridge_stp off
 599         bridge_fd 0
 600         ip-forward on
 601         ip6-forward on
 602         arp-accept on
 603 ----
 604
 605
 606 frr.conf
 607
 608 ----
 609 router bgp 1234
 610  bgp router-id 192.168.0.1
 611  no bgp default ipv4-unicast
 612  coalesce-time 1000
 613  neighbor 192.168.0.2 remote-as 1234
 614  neighbor 192.168.0.3 remote-as 1234
 615  !
 616  address-family l2vpn evpn
 617   neighbor 192.168.0.2 activate
 618   neighbor 192.168.0.3 activate
 619   advertise-all-vni
 620  exit-address-family
 621 !
 622 line vty
 623 !
 624 ----
 625
 626
 627 * node2
 628
 629 ----
 630 auto eno1
 631 iface eno1 inet manual
 632         mtu 1550
 633
 634 auto vmbr0
 635 iface vmbr0 inet static
 636          address 192.168.0.2
 637          netmask  255.255.255.0
 638          bridge_ports eno1
 639          bridge_stp off
 640          bridge_fd 0
 641
 642 auto vxlan2
 643 iface vxlan2 inet manual
 644         vxlan-id 2
 645         vxlan-local-tunnelip 192.168.0.2
 646         bridge-learning off
 647         bridge-arp-nd-suppress on
 648         bridge-unicast-flood off
 649         bridge-multicast-flood off
 650
 651
 652 auto vmbr2
 653 iface vmbr2 inet static
 654         address 10.0.2.254
 655         netmask 255.255.255.0
 656         hwaddress 44:39:39:FF:40:94
 657         bridge_ports vxlan2
 658         bridge_stp off
 659         bridge_fd 0
 660         ip-forward on
 661         ip6-forward on
 662         arp-accept on
 663
 664
 665 auto vxlan3
 666 iface vxlan3 inet manual
 667         vxlan-id 3
 668         vxlan-local-tunnelip 192.168.0.2
 669         bridge-learning off
 670         bridge-arp-nd-suppress on
 671         bridge-unicast-flood off
 672         bridge-multicast-flood off
 673
 674
 675 auto vmbr3
 676 iface vmbr3 inet static
 677         address 10.0.3.254
 678         netmask 255.255.255.0
 679         hwaddress 44:39:39:FF:40:94
 680         bridge_ports vxlan3
 681         bridge_stp off
 682         bridge_fd 0
 683         ip-forward on
 684         ip6-forward on
 685         arp-accept on
 686 ----
 687
 688
 689 frr.conf
 690
 691 ----
 692 router bgp 1234
 693  bgp router-id 192.168.0.2
 694  no bgp default ipv4-unicast
 695  coalesce-time 1000
 696  neighbor 192.168.0.1 remote-as 1234
 697  neighbor 192.168.0.3 remote-as 1234
 698  !
 699  address-family l2vpn evpn
 700   neighbor 192.168.0.1 activate
 701   neighbor 192.168.0.3 activate
 702   advertise-all-vni
 703  exit-address-family
 704 !
 705 line vty
 706 !
 707 ----
 708
 709
 710 * node3
 711
 712 ----
 713 auto eno1
 714 iface eno1 inet manual
 715         mtu 1550
 716
 717 auto vmbr0
 718 iface vmbr0 inet static
 719          address 192.168.0.3
 720          netmask  255.255.255.0
 721          bridge_ports eno1
 722          bridge_stp off
 723          bridge_fd 0
 724
 725 auto vxlan2
 726 iface vxlan2 inet manual
 727         vxlan-id 2
 728         vxlan-local-tunnelip 192.168.0.3
 729         bridge-learning off
 730         bridge-arp-nd-suppress on
 731         bridge-unicast-flood off
 732         bridge-multicast-flood off
 733
 734
 735 auto vmbr2
 736 iface vmbr2 inet static
 737         address 10.0.2.254
 738         netmask 255.255.255.0
 739         hwaddress 44:39:39:FF:40:94
 740         bridge_ports vxlan2
 741         bridge_stp off
 742         bridge_fd 0
 743         ip-forward on
 744         ip6-forward on
 745         arp-accept on
 746
 747 auto vxlan3
 748 iface vxlan3 inet manual
 749         vxlan-id 3
 750         vxlan-local-tunnelip 192.168.0.3
 751         bridge-learning off
 752         bridge-arp-nd-suppress on
 753         bridge-unicast-flood off
 754         bridge-multicast-flood off
 755
 756 auto vmbr3
 757 iface vmbr3 inet static
 758         address 10.0.3.254
 759         netmask 255.255.255.0
 760         hwaddress 44:39:39:FF:40:94
 761         bridge_ports vxlan3
 762         bridge_stp off
 763         bridge_fd 0
 764         ip-forward on
 765         ip6-forward on
 766         arp-accept on
 767 ----
 768
 769
 770 frr.conf
 771
 772 ----
 773 router bgp 1234
 774  bgp router-id 192.168.0.3
 775  no bgp default ipv4-unicast
 776  coalesce-time 1000
 777  neighbor 192.168.0.1 remote-as 1234
 778  neighbor 192.168.0.2 remote-as 1234
 779  !
 780  address-family l2vpn evpn
 781   neighbor 192.168.0.1 activate
 782   neighbor 192.168.0.2 activate
 783   advertise-all-vni
 784  exit-address-family
 785 !
 786 line vty
 787 !
 788 ----
 789
 790
 791 symmetric model
 792 ^^^^^^^^^^^^^^^
 793
 794 With this model, you don't need to have all vxlan on all nodes.
 795 This model will also be needed to route traffic to an external router.
 796
 797 The symmetric model routes and bridges on both the ingress and the egress leafs.
 798 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 799 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 800 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 801 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 802
 803 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 804
 805 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 806
 807 * node1
 808
 809 ----
 810 auto vrf1
 811 iface vrf1
 812     vrf-table auto
 813
 814 auto eno1
 815 iface eno1 inet manual
 816         mtu 1550
 817
 818 auto vmbr0
 819 iface vmbr0 inet static
 820          address 192.168.0.1
 821          netmask  255.255.255.0
 822          bridge_ports eno1
 823          bridge_stp off
 824          bridge_fd 0
 825
 826 auto vxlan2
 827 iface vxlan2 inet manual
 828         vxlan-id 2
 829         vxlan-local-tunnelip 192.168.0.1
 830         bridge-learning off
 831         bridge-arp-nd-suppress on
 832         bridge-unicast-flood off
 833         bridge-multicast-flood off
 834
 835 auto vmbr2
 836 iface vmbr2 inet static
 837         bridge_ports vxlan2
 838         bridge_stp off
 839         bridge_fd 0
 840         address 10.0.2.254
 841         netmask 255.255.255.0
 842         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 843         vrf vrf1
 844         ip-forward on
 845         ip6-forward on
 846         arp-accept on
 847
 848 auto vxlan3
 849 iface vxlan3 inet manual
 850         vxlan-id 3
 851         vxlan-local-tunnelip 192.168.0.1
 852         bridge-learning off
 853         bridge-arp-nd-suppress on
 854         bridge-unicast-flood off
 855         bridge-multicast-flood off
 856
 857 auto vmbr3
 858 iface vmbr3 inet static
 859         bridge_ports vxlan3
 860         bridge_stp off
 861         bridge_fd 0
 862         address 10.0.3.254
 863         netmask 255.255.255.0
 864         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 865         vrf vrf1
 866         ip-forward on
 867         ip6-forward on
 868         arp-accept on
 869
 870 #interconnect vxlan-vfr l3vni
 871 auto vxlan4000
 872 iface vxlan4000 inet manual
 873         vxlan-id 4000
 874         vxlan-local-tunnelip 192.168.0.1
 875         bridge-learning off
 876         bridge-arp-nd-suppress on
 877         bridge-unicast-flood off
 878         bridge-multicast-flood off
 879
 880
 881 auto vmbr4000
 882 iface vmbr4000 inet manual
 883         bridge_ports vxlan4000
 884         bridge_stp off
 885         bridge_fd 0
 886         vrf vrf1
 887 ----
 888
 889 frr.conf
 890
 891 ----
 892 vrf vrf1
 893  vni 4000
 894  exit-vrf
 895 !
 896 router bgp 1234
 897  bgp router-id 192.168.0.1
 898  no bgp default ipv4-unicast
 899  coalesce-time 1000
 900  neighbor 192.168.0.2 remote-as 1234
 901  neighbor 192.168.0.3 remote-as 1234
 902  !
 903  address-family l2vpn evpn
 904   neighbor 192.168.0.2 activate
 905   neighbor 192.168.0.3 activate
 906   advertise-all-vni
 907  exit-address-family
 908 !
 909 line vty
 910 !
 911 ----
 912
 913
 914 * node2
 915
 916 ----
 917 auto vrf1
 918 iface vrf1
 919     vrf-table auto
 920
 921 auto eno1
 922 iface eno1 inet manual
 923         mtu 1550
 924
 925 auto vmbr0
 926 iface vmbr0 inet static
 927          address 192.168.0.2
 928          netmask  255.255.255.0
 929          bridge_ports eno1
 930          bridge_stp off
 931          bridge_fd 0
 932
 933 auto vxlan2
 934 iface vxlan2 inet manual
 935         vxlan-id 2
 936         vxlan-local-tunnelip 192.168.0.2
 937         bridge-learning off
 938         bridge-arp-nd-suppress on
 939         bridge-unicast-flood off
 940         bridge-multicast-flood off
 941
 942 auto vmbr2
 943 iface vmbr2 inet static
 944         bridge_ports vxlan2
 945         bridge_stp off
 946         bridge_fd 0
 947         address 10.0.2.254
 948         netmask 255.255.255.0
 949         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 950         vrf vrf1
 951         ip-forward on
 952         ip6-forward on
 953         arp-accept on
 954
 955 auto vxlan3
 956 iface vxlan3 inet manual
 957         vxlan-id 3
 958         vxlan-local-tunnelip 192.168.0.2
 959         bridge-learning off
 960         bridge-arp-nd-suppress on
 961         bridge-unicast-flood off
 962         bridge-multicast-flood off
 963
 964 auto vmbr3
 965 iface vmbr3 inet static
 966         bridge_ports vxlan3
 967         bridge_stp off
 968         bridge_fd 0
 969         address 10.0.3.254
 970         netmask 255.255.255.0
 971         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 972         vrf vrf1
 973         ip-forward on
 974         ip6-forward on
 975         arp-accept on
 976
 977 #interconnect vxlan-vfr l3vni
 978 auto vxlan4000
 979 iface vxlan4000 inet manual
 980         vxlan-id 4000
 981         vxlan-local-tunnelip 192.168.0.2
 982         bridge-learning off
 983         bridge-arp-nd-suppress on
 984         bridge-unicast-flood off
 985         bridge-multicast-flood off
 986
 987
 988 auto vmbr4000
 989 iface vmbr4000 inet manual
 990         bridge_ports vxlan4000
 991         bridge_stp off
 992         bridge_fd 0
 993         vrf vrf1
 994 ----
 995
 996
 997 frr.conf
 998
 999 ----
1000 vrf vrf1
1001  vni 4000
1002  exit-vrf
1003 !
1004 router bgp 1234
1005  bgp router-id 192.168.0.2
1006  no bgp default ipv4-unicast
1007  coalesce-time 1000
1008  neighbor 192.168.0.1 remote-as 1234
1009  neighbor 192.168.0.3 remote-as 1234
1010  !
1011  address-family l2vpn evpn
1012   neighbor 192.168.0.1 activate
1013   neighbor 192.168.0.3 activate
1014   advertise-all-vni
1015  exit-address-family
1016 !
1017 line vty
1018 !
1019 ----
1020
1021
1022 * node3
1023
1024 ----
1025 auto vrf1
1026 iface vrf1
1027     vrf-table auto
1028
1029 auto eno1
1030 iface eno1 inet manual
1031         mtu 1550
1032
1033 auto vmbr0
1034 iface vmbr0 inet static
1035          address 192.168.0.3
1036          netmask  255.255.255.0
1037          bridge_ports eno1
1038          bridge_stp off
1039          bridge_fd 0
1040
1041 auto vxlan2
1042 iface vxlan2 inet manual
1043         vxlan-id 2
1044         vxlan-local-tunnelip 192.168.0.3
1045         bridge-learning off
1046         bridge-arp-nd-suppress on
1047         bridge-unicast-flood off
1048         bridge-multicast-flood off
1049
1050 auto vmbr2
1051 iface vmbr2 inet static
1052         bridge_ports vxlan2
1053         bridge_stp off
1054         bridge_fd 0
1055         address 10.0.2.254
1056         netmask 255.255.255.0
1057         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1058         vrf vrf1
1059         ip-forward on
1060         ip6-forward on
1061         arp-accept on
1062
1063 auto vxlan3
1064 iface vxlan3 inet manual
1065         vxlan-id 3
1066         vxlan-local-tunnelip 192.168.0.3
1067         bridge-learning off
1068         bridge-arp-nd-suppress on
1069         bridge-unicast-flood off
1070         bridge-multicast-flood off
1071
1072 auto vmbr3
1073 iface vmbr3 inet static
1074         bridge_ports vxlan3
1075         bridge_stp off
1076         bridge_fd 0
1077         address 10.0.3.254
1078         netmask 255.255.255.0
1079         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1080         vrf vrf1
1081         ip-forward on
1082         ip6-forward on
1083         arp-accept on
1084
1085 #interconnect vxlan-vfr l3vni
1086 auto vxlan4000
1087 iface vxlan4000 inet manual
1088         vxlan-id 4000
1089         vxlan-local-tunnelip 192.168.0.3
1090         bridge-learning off
1091         bridge-arp-nd-suppress on
1092         bridge-unicast-flood off
1093         bridge-multicast-flood off
1094
1095
1096 auto vmbr4000
1097 iface vmbr4000 inet manual
1098         bridge_ports vxlan4000
1099         bridge_stp off
1100         bridge_fd 0
1101         vrf vrf1
1102 ----
1103
1104
1105 frr.conf
1106
1107 ----
1108 vrf vrf1
1109  vni 4000
1110  exit-vrf
1111 !
1112 router bgp 1234
1113  bgp router-id 192.168.0.3
1114  no bgp default ipv4-unicast
1115  coalesce-time 1000
1116  neighbor 192.168.0.1 remote-as 1234
1117  neighbor 192.168.0.2 remote-as 1234
1118  !
1119  address-family l2vpn evpn
1120   neighbor 192.168.0.1 activate
1121   neighbor 192.168.0.2 activate
1122   advertise-all-vni
1123  exit-address-family
1124 !
1125 line vty
1126 !
1127 ----
1128
1129 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1130 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1131 Routing to outside need the symmetric model.
1132
1133 1 gateway node
1134 ^^^^^^^^^^^^^^
1135 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1136 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1137
1138
1139 *node1
1140
1141 ----
1142 auto vrf1
1143 iface vrf1
1144     vrf-table auto
1145
1146 auto eno1
1147 iface eno1 inet manual
1148         mtu 1550
1149
1150 auto vmbr0
1151 iface vmbr0 inet static
1152          address 192.168.0.1
1153          netmask  255.255.255.0
1154          gateway 192.168.0.254
1155          bridge_ports eno1
1156          bridge_stp off
1157          bridge_fd 0
1158          ip-forward on
1159          ip6-forward on
1160
1161 auto vxlan2
1162 iface vxlan2 inet manual
1163         vxlan-id 2
1164         vxlan-local-tunnelip 192.168.0.1
1165         bridge-learning off
1166         bridge-arp-nd-suppress on
1167         bridge-unicast-flood off
1168         bridge-multicast-flood off
1169
1170 auto vmbr2
1171 iface vmbr2 inet static
1172         bridge_ports vxlan2
1173         bridge_stp off
1174         bridge_fd 0
1175         address 10.0.2.254
1176         netmask 255.255.255.0
1177         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1178         vrf vrf1
1179         ip-forward on
1180         ip6-forward on
1181         arp-accept on
1182
1183 auto vxlan3
1184 iface vxlan3 inet manual
1185         vxlan-id 3
1186         vxlan-local-tunnelip 192.168.0.1
1187         bridge-learning off
1188         bridge-arp-nd-suppress on
1189         bridge-unicast-flood off
1190         bridge-multicast-flood off
1191
1192 auto vmbr3
1193 iface vmbr3 inet static
1194         bridge_ports vxlan3
1195         bridge_stp off
1196         bridge_fd 0
1197         address 10.0.3.254
1198         netmask 255.255.255.0
1199         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1200         vrf vrf1
1201         ip-forward on
1202         ip6-forward on
1203         arp-accept on
1204
1205 #interconnect vxlan-vfr l3vni
1206 auto vxlan4000
1207 iface vxlan4000 inet manual
1208         vxlan-id 4000
1209         vxlan-local-tunnelip 192.168.0.1
1210         bridge-learning off
1211         bridge-arp-nd-suppress on
1212         bridge-unicast-flood off
1213         bridge-multicast-flood off
1214
1215 auto vmbr4000
1216 iface vmbr4000 inet manual
1217         bridge_ports vxlan4000
1218         bridge_stp off
1219         bridge_fd 0
1220         vrf vrf1
1221 ----
1222
1223
1224 frr.conf
1225
1226 ----
1227 ip prefix-list deny seq 10 deny any
1228 !
1229 vrf vrf1
1230  vni 4000
1231  exit-vrf
1232 !
1233 router bgp 1234
1234  bgp router-id 192.168.0.1
1235  no bgp default ipv4-unicast
1236  coalesce-time 1000
1237  neighbor 192.168.0.2 remote-as 1234
1238  neighbor 192.168.0.3 remote-as 1234
1239  !
1240  address-family ipv4 unicast
1241   import vrf vrf1
1242   neighbor 192.168.0.2 prefix-list deny out
1243   neighbor 192.168.0.3 prefix-list deny out
1244  exit-address-family
1245  !
1246  address-family l2vpn evpn
1247   neighbor 192.168.0.2 activate
1248   neighbor 192.168.0.3 activate
1249   advertise-all-vni
1250  exit-address-family
1251 !
1252 router bgp 1234 vrf vrf1
1253 !
1254  address-family ipv4 unicast
1255   redistribute connected
1256  exit-address-family
1257  !
1258  address-family l2vpn evpn
1259   default-originate ipv4
1260  exit-address-family
1261 !
1262 line vty
1263 !
1264 ----
1265
1266
1267 * node2
1268
1269 ----
1270 auto vrf1
1271 iface vrf1
1272     vrf-table auto
1273
1274 auto eno1
1275 iface eno1 inet manual
1276         mtu 1550
1277
1278 auto vmbr0
1279 iface vmbr0 inet static
1280          address 192.168.0.2
1281          netmask  255.255.255.0
1282          bridge_ports eno1
1283          bridge_stp off
1284          bridge_fd 0
1285
1286 auto vxlan2
1287 iface vxlan2 inet manual
1288         vxlan-id 2
1289         vxlan-local-tunnelip 192.168.0.2
1290         bridge-learning off
1291         bridge-arp-nd-suppress on
1292         bridge-unicast-flood off
1293         bridge-multicast-flood off
1294
1295 auto vmbr2
1296 iface vmbr2 inet static
1297         bridge_ports vxlan2
1298         bridge_stp off
1299         bridge_fd 0
1300         address 10.0.2.254
1301         netmask 255.255.255.0
1302         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1303         vrf vrf1
1304         ip-forward on
1305         ip6-forward on
1306         arp-accept on
1307
1308 auto vxlan3
1309 iface vxlan3 inet manual
1310         vxlan-id 3
1311         vxlan-local-tunnelip 192.168.0.2
1312         bridge-learning off
1313         bridge-arp-nd-suppress on
1314         bridge-unicast-flood off
1315         bridge-multicast-flood off
1316
1317 auto vmbr3
1318 iface vmbr3 inet static
1319         bridge_ports vxlan3
1320         bridge_stp off
1321         bridge_fd 0
1322         address 10.0.3.254
1323         netmask 255.255.255.0
1324         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1325         vrf vrf1
1326         ip-forward on
1327         ip6-forward on
1328         arp-accept on
1329
1330 #interconnect vxlan-vfr l3vni
1331 auto vxlan4000
1332 iface vxlan4000 inet manual
1333         vxlan-id 4000
1334         vxlan-local-tunnelip 192.168.0.2
1335         bridge-learning off
1336         bridge-arp-nd-suppress on
1337         bridge-unicast-flood off
1338         bridge-multicast-flood off
1339
1340
1341 auto vmbr4000
1342 iface vmbr4000 inet manual
1343         bridge_ports vxlan4000
1344         bridge_stp off
1345         bridge_fd 0
1346         vrf vrf1
1347 ----
1348
1349
1350 frr.conf
1351
1352 ----
1353 vrf vrf1
1354  vni 4000
1355  exit-vrf
1356 !
1357 router bgp 1234
1358  bgp router-id 192.168.0.2
1359  no bgp default ipv4-unicast
1360  coalesce-time 1000
1361  neighbor 192.168.0.1 remote-as 1234
1362  neighbor 192.168.0.3 remote-as 1234
1363  !
1364  address-family l2vpn evpn
1365   neighbor 192.168.0.1 activate
1366   neighbor 192.168.0.3 activate
1367   advertise-all-vni
1368  exit-address-family
1369 !
1370 line vty
1371 !
1372 ----
1373
1374
1375 * node3
1376
1377 ----
1378 auto vrf1
1379 iface vrf1
1380     vrf-table auto
1381
1382 auto eno1
1383 iface eno1 inet manual
1384         mtu 1550
1385
1386 auto vmbr0
1387 iface vmbr0 inet static
1388          address 192.168.0.3
1389          netmask  255.255.255.0
1390          bridge_ports eno1
1391          bridge_stp off
1392          bridge_fd 0
1393
1394 auto vxlan2
1395 iface vxlan2 inet manual
1396         vxlan-id 2
1397         vxlan-local-tunnelip 192.168.0.3
1398         bridge-learning off
1399         bridge-arp-nd-suppress on
1400         bridge-unicast-flood off
1401         bridge-multicast-flood off
1402
1403 auto vmbr2
1404 iface vmbr2 inet static
1405         bridge_ports vxlan2
1406         bridge_stp off
1407         bridge_fd 0
1408         address 10.0.2.254
1409         netmask 255.255.255.0
1410         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1411         vrf vrf1
1412         ip-forward on
1413         ip6-forward on
1414         arp-accept on
1415
1416 auto vxlan3
1417 iface vxlan3 inet manual
1418         vxlan-id 3
1419         vxlan-local-tunnelip 192.168.0.3
1420         bridge-learning off
1421         bridge-arp-nd-suppress on
1422         bridge-unicast-flood off
1423         bridge-multicast-flood off
1424
1425 auto vmbr3
1426 iface vmbr3 inet static
1427         bridge_ports vxlan3
1428         bridge_stp off
1429         bridge_fd 0
1430         address 10.0.3.254
1431         netmask 255.255.255.0
1432         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1433         vrf vrf1
1434         ip-forward on
1435         ip6-forward on
1436         arp-accept on
1437
1438 #interconnect vxlan-vfr l3vni
1439 auto vxlan4000
1440 iface vxlan4000 inet manual
1441         vxlan-id 4000
1442         vxlan-local-tunnelip 192.168.0.3
1443         bridge-learning off
1444         bridge-arp-nd-suppress on
1445         bridge-unicast-flood off
1446         bridge-multicast-flood off
1447
1448
1449 auto vmbr4000
1450 iface vmbr4000 inet manual
1451         bridge_ports vxlan4000
1452         bridge_stp off
1453         bridge_fd 0
1454         vrf vrf1
1455 ----
1456
1457
1458 frr.conf
1459
1460 ----
1461 vrf vrf1
1462  vni 4000
1463  exit-vrf
1464 !
1465 router bgp 1234
1466  bgp router-id 192.168.0.3
1467  no bgp default ipv4-unicast
1468  coalesce-time 1000
1469  neighbor 192.168.0.1 remote-as 1234
1470  neighbor 192.168.0.2 remote-as 1234
1471  !
1472  address-family l2vpn evpn
1473   neighbor 192.168.0.1 activate
1474   neighbor 192.168.0.2 activate
1475   advertise-all-vni
1476  exit-address-family
1477 !
1478 line vty
1479 !
1480 ----
1481
1482 multiple gateway nodes
1483 ^^^^^^^^^^^^^^^^^^^^^^
1484 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1485 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1486 and announce this default gw in the vrf (default originate)
1487 The external router have ecmp routes to all proxmox nodes.(balancing).
1488 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1489 vxlan the packet to final destination.
1490
1491 *node1
1492
1493 ----
1494 auto vrf1
1495 iface vrf1
1496     vrf-table auto
1497
1498 auto eno1
1499 iface eno1 inet manual
1500         mtu 1550
1501
1502 auto vmbr0
1503 iface vmbr0 inet static
1504          address 192.168.0.1
1505          netmask  255.255.255.0
1506          gateway 192.168.0.254
1507          bridge_ports eno1
1508          bridge_stp off
1509          bridge_fd 0
1510          ip-forward on
1511          ip6-forward on
1512
1513 auto vxlan2
1514 iface vxlan2 inet manual
1515         vxlan-id 2
1516         vxlan-local-tunnelip 192.168.0.1
1517         bridge-learning off
1518         bridge-arp-nd-suppress on
1519         bridge-unicast-flood off
1520         bridge-multicast-flood off
1521
1522 auto vmbr2
1523 iface vmbr2 inet static
1524         bridge_ports vxlan2
1525         bridge_stp off
1526         bridge_fd 0
1527         address 10.0.2.254
1528         netmask 255.255.255.0
1529         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1530         vrf vrf1
1531         ip-forward on
1532         ip6-forward on
1533         arp-accept on
1534
1535 auto vxlan3
1536 iface vxlan3 inet manual
1537         vxlan-id 3
1538         vxlan-local-tunnelip 192.168.0.1
1539         bridge-learning off
1540         bridge-arp-nd-suppress on
1541         bridge-unicast-flood off
1542         bridge-multicast-flood off
1543
1544 auto vmbr3
1545 iface vmbr3 inet static
1546         bridge_ports vxlan3
1547         bridge_stp off
1548         bridge_fd 0
1549         address 10.0.3.254
1550         netmask 255.255.255.0
1551         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1552         vrf vrf1
1553         ip-forward on
1554         ip6-forward on
1555         arp-accept on
1556
1557 #interconnect vxlan-vfr l3vni
1558 auto vxlan4000
1559 iface vxlan4000 inet manual
1560         vxlan-id 4000
1561         vxlan-local-tunnelip 192.168.0.1
1562         bridge-learning off
1563         bridge-arp-nd-suppress on
1564         bridge-unicast-flood off
1565         bridge-multicast-flood off
1566
1567 auto vmbr4000
1568 iface vmbr4000 inet manual
1569         bridge_ports vxlan4000
1570         bridge_stp off
1571         bridge_fd 0
1572         vrf vrf1
1573 ----
1574
1575
1576 frr.conf
1577
1578 ----
1579 ip prefix-list deny seq 10 deny any
1580 !
1581 vrf vrf1
1582  vni 4000
1583  exit-vrf
1584 !
1585 router bgp 1234
1586  bgp router-id 192.168.0.1
1587  no bgp default ipv4-unicast
1588  coalesce-time 1000
1589  neighbor 192.168.0.2 remote-as 1234
1590  neighbor 192.168.0.3 remote-as 1234
1591  !
1592  address-family ipv4 unicast
1593   import vrf vrf1
1594   neighbor 192.168.0.2 prefix-list deny out
1595   neighbor 192.168.0.3 prefix-list deny out
1596  exit-address-family
1597  !
1598  address-family l2vpn evpn
1599   neighbor 192.168.0.2 activate
1600   neighbor 192.168.0.3 activate
1601   advertise-all-vni
1602  exit-address-family
1603 !
1604 router bgp 1234 vrf vrf1
1605 !
1606  address-family ipv4 unicast
1607   redistribute connected
1608  exit-address-family
1609  !
1610  address-family l2vpn evpn
1611   default-originate ipv4
1612  exit-address-family
1613 !
1614 line vty
1615 !
1616 ----
1617
1618
1619 * node2
1620
1621 ----
1622 auto vrf1
1623 iface vrf1
1624     vrf-table auto
1625
1626 auto eno1
1627 iface eno1 inet manual
1628         mtu 1550
1629
1630 auto vmbr0
1631 iface vmbr0 inet static
1632          address 192.168.0.2
1633          netmask  255.255.255.0
1634          gateway 192.168.0.254
1635          bridge_ports eno1
1636          bridge_stp off
1637          bridge_fd 0
1638          ip-forward on
1639          ip6-forward on
1640
1641 auto vxlan2
1642 iface vxlan2 inet manual
1643         vxlan-id 2
1644         vxlan-local-tunnelip 192.168.0.2
1645         bridge-learning off
1646         bridge-arp-nd-suppress on
1647         bridge-unicast-flood off
1648         bridge-multicast-flood off
1649
1650 auto vmbr2
1651 iface vmbr2 inet static
1652         bridge_ports vxlan2
1653         bridge_stp off
1654         bridge_fd 0
1655         address 10.0.2.254
1656         netmask 255.255.255.0
1657         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1658         vrf vrf1
1659         ip-forward on
1660         ip6-forward on
1661         arp-accept on
1662
1663 auto vxlan3
1664 iface vxlan3 inet manual
1665         vxlan-id 3
1666         vxlan-local-tunnelip 192.168.0.2
1667         bridge-learning off
1668         bridge-arp-nd-suppress on
1669         bridge-unicast-flood off
1670         bridge-multicast-flood off
1671
1672 auto vmbr3
1673 iface vmbr3 inet static
1674         bridge_ports vxlan3
1675         bridge_stp off
1676         bridge_fd 0
1677         address 10.0.3.254
1678         netmask 255.255.255.0
1679         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1680         vrf vrf1
1681         ip-forward on
1682         ip6-forward on
1683         arp-accept on
1684
1685 #interconnect vxlan-vfr l3vni
1686 auto vxlan4000
1687 iface vxlan4000 inet manual
1688         vxlan-id 4000
1689         vxlan-local-tunnelip 192.168.0.2
1690         bridge-learning off
1691         bridge-arp-nd-suppress on
1692         bridge-unicast-flood off
1693         bridge-multicast-flood off
1694
1695
1696 auto vmbr4000
1697 iface vmbr4000 inet manual
1698         bridge_ports vxlan4000
1699         bridge_stp off
1700         bridge_fd 0
1701         vrf vrf1
1702 ----
1703
1704
1705 frr.conf
1706
1707 ----
1708 ip prefix-list deny seq 10 deny any
1709 !
1710 vrf vrf1
1711  vni 4000
1712  exit-vrf
1713 !
1714 router bgp 1234
1715  bgp router-id 192.168.0.2
1716  no bgp default ipv4-unicast
1717  coalesce-time 1000
1718  neighbor 192.168.0.1 remote-as 1234
1719  neighbor 192.168.0.3 remote-as 1234
1720  !
1721  address-family ipv4 unicast
1722   import vrf vrf1
1723   neighbor 192.168.0.1 prefix-list deny out
1724   neighbor 192.168.0.3 prefix-list deny out
1725  exit-address-family
1726  !
1727  address-family l2vpn evpn
1728   neighbor 192.168.0.1 activate
1729   neighbor 192.168.0.3 activate
1730   advertise-all-vni
1731  exit-address-family
1732 !
1733  address-family ipv4 unicast
1734   redistribute connected
1735  exit-address-family
1736  !
1737  address-family l2vpn evpn
1738   default-originate ipv4
1739  exit-address-family
1740 !
1741 line vty
1742 !
1743 ----
1744
1745
1746 * node3
1747
1748 ----
1749 auto vrf1
1750 iface vrf1
1751     vrf-table auto
1752
1753 auto eno1
1754 iface eno1 inet manual
1755         mtu 1550
1756
1757 auto vmbr0
1758 iface vmbr0 inet static
1759          address 192.168.0.3
1760          netmask  255.255.255.0
1761          gateway 192.168.0.254
1762          bridge_ports eno1
1763          bridge_stp off
1764          bridge_fd 0
1765          ip-forward on
1766          ip6-forward on
1767
1768 auto vxlan2
1769 iface vxlan2 inet manual
1770         vxlan-id 2
1771         vxlan-local-tunnelip 192.168.0.3
1772         bridge-learning off
1773         bridge-arp-nd-suppress on
1774         bridge-unicast-flood off
1775         bridge-multicast-flood off
1776
1777 auto vmbr2
1778 iface vmbr2 inet static
1779         bridge_ports vxlan2
1780         bridge_stp off
1781         bridge_fd 0
1782         address 10.0.2.254
1783         netmask 255.255.255.0
1784         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1785         vrf vrf1
1786         ip-forward on
1787         ip6-forward on
1788         arp-accept on
1789
1790 auto vxlan3
1791 iface vxlan3 inet manual
1792         vxlan-id 3
1793         vxlan-local-tunnelip 192.168.0.3
1794         bridge-learning off
1795         bridge-arp-nd-suppress on
1796         bridge-unicast-flood off
1797         bridge-multicast-flood off
1798
1799 auto vmbr3
1800 iface vmbr3 inet static
1801         bridge_ports vxlan3
1802         bridge_stp off
1803         bridge_fd 0
1804         address 10.0.3.254
1805         netmask 255.255.255.0
1806         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1807         vrf vrf1
1808         ip-forward on
1809         ip6-forward on
1810         arp-accept on
1811
1812 #interconnect vxlan-vfr l3vni
1813 auto vxlan4000
1814 iface vxlan4000 inet manual
1815         vxlan-id 4000
1816         vxlan-local-tunnelip 192.168.0.3
1817         bridge-learning off
1818         bridge-arp-nd-suppress on
1819         bridge-unicast-flood off
1820         bridge-multicast-flood off
1821
1822
1823 auto vmbr4000
1824 iface vmbr4000 inet manual
1825         bridge_ports vxlan4000
1826         bridge_stp off
1827         bridge_fd 0
1828         vrf vrf1
1829 ----
1830
1831
1832 frr.conf
1833
1834 ----
1835 ip prefix-list deny seq 10 deny any
1836 !
1837 vrf vrf1
1838  vni 4000
1839  exit-vrf
1840 !
1841 router bgp 1234
1842  bgp router-id 192.168.0.3
1843  no bgp default ipv4-unicast
1844  coalesce-time 1000
1845  neighbor 192.168.0.1 remote-as 1234
1846  neighbor 192.168.0.2 remote-as 1234
1847  !
1848  address-family ipv4 unicast
1849   import vrf vrf1
1850   neighbor 192.168.0.1 prefix-list deny out
1851   neighbor 192.168.0.2 prefix-list deny out
1852  exit-address-family
1853  !
1854  address-family l2vpn evpn
1855   neighbor 192.168.0.1 activate
1856   neighbor 192.168.0.2 activate
1857   advertise-all-vni
1858  exit-address-family
1859 !
1860 router bgp 1234 vrf vrf1
1861 !
1862  address-family ipv4 unicast
1863   redistribute connected
1864  exit-address-family
1865  !
1866  address-family l2vpn evpn
1867   default-originate ipv4
1868  exit-address-family
1869 !
1870 line vty
1871 !
1872 ----
1873
1874 Note
1875 ^^^^
1876
1877 If your external router doesn't support 'ECMP static routes' to reach multiple
1878 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1879 Virtual Router Redundancy Protocol (VRRP).
1880
1881 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1882 Node1 is the primary with failover to node2 in case of outage.
1883
1884 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1885 #TODO : It should be possible to do it with frr directly with last version.
1886
1887 * node1
1888
1889 ----
1890 auto vmbr0
1891 iface vmbr0 inet static
1892          address 192.168.0.1
1893          netmask  255.255.255.0
1894          gateway 192.168.0.254
1895          bridge_ports eno1
1896          bridge_stp off
1897          bridge_fd 0
1898          vrrp-id 1
1899          vrrp-priority 1
1900          vrrp-virtual-ip 192.168.0.10
1901 ----
1902
1903 * node2
1904
1905 ----
1906 auto vmbr0
1907 iface vmbr0 inet static
1908          address 192.168.0.2
1909          netmask  255.255.255.0
1910          gateway 192.168.0.254
1911          bridge_ports eno1
1912          bridge_stp off
1913          bridge_fd 0
1914          vrrp-id 1
1915          vrrp-priority 2
1916          vrrp-virtual-ip 192.168.0.10
1917 ----
1918
1919
1920 Route Reflectors
1921 ^^^^^^^^^^^^^^^^
1922 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
1923 to avoid that all node peers with each others nodes.
1924 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
1925 single point of failure, a minimum of two servers acting as an RR is highly
1926 recommended for redundancy.
1927
1928 Below is an example of configuration with 'frr', with `rrserver1
1929 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
1930
1931 rrserver1
1932 ----
1933 router bgp 1234
1934   bgp router-id 192.168.0.200
1935   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
1936   bgp log-neighbor-changes
1937   no bgp default ipv4-unicast
1938   neighbor fabric peer-group
1939   neighbor fabric remote-as 1234
1940   neighbor fabric capability extended-nexthop
1941   neighbor fabric update-source 192.168.0.200
1942   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
1943   !
1944   address-family l2vpn evpn
1945    neighbor fabric activate
1946    neighbor fabric route-reflector-client
1947    neighbor fabric allowas-in
1948   exit-address-family
1949   !
1950   exit
1951 !
1952 ---
1953
1954 rrserver2
1955 ----
1956 router bgp 1234
1957   bgp router-id 192.168.0.201
1958   bgp cluster-id 1.1.1.1
1959   bgp log-neighbor-changes
1960   no bgp default ipv4-unicast
1961   neighbor fabric peer-group
1962   neighbor fabric remote-as 1234
1963   neighbor fabric capability extended-nexthop
1964   neighbor fabric update-source 192.168.0.201
1965   bgp listen range 192.168.0.0/24 peer-group fabric
1966   !
1967   address-family l2vpn evpn
1968    neighbor fabric activate
1969    neighbor fabric route-reflector-client
1970    neighbor fabric allowas-in
1971   exit-address-family
1972   !
1973   exit
1974 !
1975 ---
1976
1977 proxmoxnode(s)
1978 ----
1979 router bgp 1234
1980  bgp router-id 192.168.0.x
1981  no bgp default ipv4-unicast
1982  coalesce-time 1000
1983  neighbor 192.168.0.200 remote-as 1234
1984  neighbor 192.168.0.201 remote-as 1234
1985  !
1986  address-family l2vpn evpn
1987   neighbor 192.168.0.200 activate
1988   neighbor 192.168.0.201 activate
1989   advertise-all-vni
1990  exit-address-family
1991 !
1992 ----
1993
1994 #TODO : Documentation with bgp upstream router.