vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  no bgp default ipv6-unicast
 371  coalesce-time 1000
 372  neighbor 192.168.0.2 remote-as 1234
 373  neighbor 192.168.0.3 remote-as 1234
 374  !
 375  address-family l2vpn evpn
 376   neighbor 192.168.0.2 activate
 377   neighbor 192.168.0.3 activate
 378   advertise-all-vni
 379  exit-address-family
 380 !
 381 line vty
 382 !
 383 ----
 384
 385
 386 * node2
 387
 388 ----
 389 auto eno1
 390 iface eno1 inet manual
 391         mtu 1550
 392
 393 auto vmbr0
 394 iface vmbr0 inet static
 395         address 192.168.0.2
 396         netmask 255.255.255.0
 397         bridge_ports eno1
 398         bridge_stp off
 399         bridge_fd 0
 400
 401 auto vxlan2
 402 iface vxlan2 inet manual
 403         vxlan-id 2
 404         vxlan-local-tunnelip 192.168.0.2
 405         bridge-learning off
 406         bridge-arp-nd-suppress on
 407         bridge-unicast-flood off
 408         bridge-multicast-flood off
 409
 410
 411 auto vmbr2
 412 iface vmbr2 inet manual
 413         bridge_ports vxlan2
 414         bridge_stp off
 415         bridge_fd 0
 416
 417 auto vxlan3
 418 iface vxlan3 inet manual
 419         vxlan-id 3
 420         vxlan-local-tunnelip 192.168.0.2
 421         bridge-learning off
 422         bridge-arp-nd-suppress on
 423         bridge-unicast-flood off
 424         bridge-multicast-flood off
 425
 426
 427 auto vmbr3
 428 iface vmbr3 inet manual
 429         bridge_ports vxlan3
 430         bridge_stp off
 431         bridge_fd 0
 432 ----
 433
 434
 435 /etc/frr/frr.conf
 436
 437 ----
 438 router bgp 1234
 439  no bgp default ipv4-unicast
 440  no bgp default ipv6-unicast
 441  coalesce-time 1000
 442  neighbor 192.168.0.1 remote-as 1234
 443  neighbor 192.168.0.3 remote-as 1234
 444  !
 445  address-family l2vpn evpn
 446   neighbor 192.168.0.1 activate
 447   neighbor 192.168.0.3 activate
 448   advertise-all-vni
 449  exit-address-family
 450 !
 451 line vty
 452 !
 453 ----
 454
 455
 456 * node3
 457
 458 ----
 459 auto eno1
 460 iface eno1 inet manual
 461         mtu 1550
 462
 463 auto vmbr0
 464 iface vmbr0 inet static
 465         address 192.168.0.2
 466         netmask 255.255.255.0
 467         bridge_ports eno1
 468         bridge_stp off
 469         bridge_fd 0
 470
 471 auto vxlan2
 472 iface vxlan2 inet manual
 473         vxlan-id 2
 474         vxlan-local-tunnelip 192.168.0.3
 475         bridge-learning off
 476         bridge-arp-nd-suppress on
 477         bridge-unicast-flood off
 478         bridge-multicast-flood off
 479
 480
 481 auto vmbr2
 482 iface vmbr2 inet manual
 483         bridge_ports vxlan2
 484         bridge_stp off
 485         bridge_fd 0
 486
 487 auto vxlan3
 488 iface vxlan3 inet manual
 489         vxlan-id 3
 490         vxlan-local-tunnelip 192.168.0.3
 491         bridge-learning off
 492         bridge-arp-nd-suppress on
 493         bridge-unicast-flood off
 494         bridge-multicast-flood off
 495
 496
 497 auto vmbr3
 498 iface vmbr3 inet manual
 499         bridge_ports vxlan3
 500         bridge_stp off
 501         bridge_fd 0
 502 ----
 503
 504
 505 /etc/frr/frr.conf
 506
 507
 508 ----
 509 router bgp 1234
 510  no bgp default ipv4-unicast
 511  no bgp default ipv6-unicast
 512  coalesce-time 1000
 513  neighbor 192.168.0.1 remote-as 1234
 514  neighbor 192.168.0.2 remote-as 1234
 515  !
 516  address-family l2vpn evpn
 517   neighbor 192.168.0.1 activate
 518   neighbor 192.168.0.2 activate
 519   advertise-all-vni
 520  exit-address-family
 521 !
 522 line vty
 523 !
 524 ----
 525
 526 VXLAN layer3 routing with anycast gateway
 527 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 528
 529 With this need, each vmbr bridge will be the gateway for the vm.
 530 Same vmbr on different node, will have same ip address and same mac address,
 531 to have working vm live migration and no network disruption.
 532
 533 VXLAN layer3 routing only work with FRR and non-aware bridge.
 534 (vlan aware bridge support is buggy currently).
 535
 536 asymmetric model
 537 ^^^^^^^^^^^^^^^^
 538
 539 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 540
 541 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 542 but only bridging on the egress.
 543 This results in bi-directional VXLAN traffic traveling on different VNIs
 544 in each direction (always the destination VNI) across the routed infrastructure.
 545
 546 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 547
 548 * node1
 549
 550 ----
 551 auto eno1
 552 iface eno1 inet manual
 553         mtu 1550
 554
 555 auto vmbr0
 556 iface vmbr0 inet static
 557          address 192.168.0.1
 558          netmask  255.255.255.0
 559          bridge_ports eno1
 560          bridge_stp off
 561          bridge_fd 0
 562
 563 auto vxlan2
 564 iface vxlan2 inet manual
 565         vxlan-id 2
 566         vxlan-local-tunnelip 192.168.0.1
 567         bridge-learning off
 568         bridge-arp-nd-suppress on
 569         bridge-unicast-flood off
 570         bridge-multicast-flood off
 571
 572
 573 auto vmbr2
 574 iface vmbr2 inet static
 575         address 10.0.2.254
 576         netmask 255.255.255.0
 577         hwaddress 44:39:39:FF:40:94
 578         bridge_ports vxlan2
 579         bridge_stp off
 580         bridge_fd 0
 581         ip-forward on
 582         ip6-forward on
 583         arp-accept on
 584
 585 auto vxlan3
 586 iface vxlan3 inet manual
 587         vxlan-id 3
 588         vxlan-local-tunnelip 192.168.0.1
 589         bridge-learning off
 590         bridge-arp-nd-suppress on
 591         bridge-unicast-flood off
 592         bridge-multicast-flood off
 593
 594
 595 auto vmbr3
 596 iface vmbr3 inet static
 597         address 10.0.3.254
 598         netmask 255.255.255.0
 599         hwaddress 44:39:39:FF:40:94
 600         bridge_ports vxlan3
 601         bridge_stp off
 602         bridge_fd 0
 603         ip-forward on
 604         ip6-forward on
 605         arp-accept on
 606 ----
 607
 608
 609 frr.conf
 610
 611 ----
 612 router bgp 1234
 613  bgp router-id 192.168.0.1
 614  no bgp default ipv4-unicast
 615  no bgp default ipv6-unicast
 616  coalesce-time 1000
 617  neighbor 192.168.0.2 remote-as 1234
 618  neighbor 192.168.0.3 remote-as 1234
 619  !
 620  address-family l2vpn evpn
 621   neighbor 192.168.0.2 activate
 622   neighbor 192.168.0.3 activate
 623   advertise-all-vni
 624  exit-address-family
 625 !
 626 line vty
 627 !
 628 ----
 629
 630
 631 * node2
 632
 633 ----
 634 auto eno1
 635 iface eno1 inet manual
 636         mtu 1550
 637
 638 auto vmbr0
 639 iface vmbr0 inet static
 640          address 192.168.0.2
 641          netmask  255.255.255.0
 642          bridge_ports eno1
 643          bridge_stp off
 644          bridge_fd 0
 645
 646 auto vxlan2
 647 iface vxlan2 inet manual
 648         vxlan-id 2
 649         vxlan-local-tunnelip 192.168.0.2
 650         bridge-learning off
 651         bridge-arp-nd-suppress on
 652         bridge-unicast-flood off
 653         bridge-multicast-flood off
 654
 655
 656 auto vmbr2
 657 iface vmbr2 inet static
 658         address 10.0.2.254
 659         netmask 255.255.255.0
 660         hwaddress 44:39:39:FF:40:94
 661         bridge_ports vxlan2
 662         bridge_stp off
 663         bridge_fd 0
 664         ip-forward on
 665         ip6-forward on
 666         arp-accept on
 667
 668
 669 auto vxlan3
 670 iface vxlan3 inet manual
 671         vxlan-id 3
 672         vxlan-local-tunnelip 192.168.0.2
 673         bridge-learning off
 674         bridge-arp-nd-suppress on
 675         bridge-unicast-flood off
 676         bridge-multicast-flood off
 677
 678
 679 auto vmbr3
 680 iface vmbr3 inet static
 681         address 10.0.3.254
 682         netmask 255.255.255.0
 683         hwaddress 44:39:39:FF:40:94
 684         bridge_ports vxlan3
 685         bridge_stp off
 686         bridge_fd 0
 687         ip-forward on
 688         ip6-forward on
 689         arp-accept on
 690 ----
 691
 692
 693 frr.conf
 694
 695 ----
 696 router bgp 1234
 697  bgp router-id 192.168.0.2
 698  no bgp default ipv4-unicast
 699  no bgp default ipv6-unicast
 700  coalesce-time 1000
 701  neighbor 192.168.0.1 remote-as 1234
 702  neighbor 192.168.0.3 remote-as 1234
 703  !
 704  address-family l2vpn evpn
 705   neighbor 192.168.0.1 activate
 706   neighbor 192.168.0.3 activate
 707   advertise-all-vni
 708  exit-address-family
 709 !
 710 line vty
 711 !
 712 ----
 713
 714
 715 * node3
 716
 717 ----
 718 auto eno1
 719 iface eno1 inet manual
 720         mtu 1550
 721
 722 auto vmbr0
 723 iface vmbr0 inet static
 724          address 192.168.0.3
 725          netmask  255.255.255.0
 726          bridge_ports eno1
 727          bridge_stp off
 728          bridge_fd 0
 729
 730 auto vxlan2
 731 iface vxlan2 inet manual
 732         vxlan-id 2
 733         vxlan-local-tunnelip 192.168.0.3
 734         bridge-learning off
 735         bridge-arp-nd-suppress on
 736         bridge-unicast-flood off
 737         bridge-multicast-flood off
 738
 739
 740 auto vmbr2
 741 iface vmbr2 inet static
 742         address 10.0.2.254
 743         netmask 255.255.255.0
 744         hwaddress 44:39:39:FF:40:94
 745         bridge_ports vxlan2
 746         bridge_stp off
 747         bridge_fd 0
 748         ip-forward on
 749         ip6-forward on
 750         arp-accept on
 751
 752 auto vxlan3
 753 iface vxlan3 inet manual
 754         vxlan-id 3
 755         vxlan-local-tunnelip 192.168.0.3
 756         bridge-learning off
 757         bridge-arp-nd-suppress on
 758         bridge-unicast-flood off
 759         bridge-multicast-flood off
 760
 761 auto vmbr3
 762 iface vmbr3 inet static
 763         address 10.0.3.254
 764         netmask 255.255.255.0
 765         hwaddress 44:39:39:FF:40:94
 766         bridge_ports vxlan3
 767         bridge_stp off
 768         bridge_fd 0
 769         ip-forward on
 770         ip6-forward on
 771         arp-accept on
 772 ----
 773
 774
 775 frr.conf
 776
 777 ----
 778 router bgp 1234
 779  bgp router-id 192.168.0.3
 780  no bgp default ipv4-unicast
 781  no bgp default ipv6-unicast
 782  coalesce-time 1000
 783  neighbor 192.168.0.1 remote-as 1234
 784  neighbor 192.168.0.2 remote-as 1234
 785  !
 786  address-family l2vpn evpn
 787   neighbor 192.168.0.1 activate
 788   neighbor 192.168.0.2 activate
 789   advertise-all-vni
 790  exit-address-family
 791 !
 792 line vty
 793 !
 794 ----
 795
 796
 797 symmetric model
 798 ^^^^^^^^^^^^^^^
 799
 800 With this model, you don't need to have all vxlan on all nodes.
 801 This model will also be needed to route traffic to an external router.
 802
 803 The symmetric model routes and bridges on both the ingress and the egress leafs.
 804 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 805 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 806 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 807 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 808
 809 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 810
 811 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 812
 813 * node1
 814
 815 ----
 816 auto vrf1
 817 iface vrf1
 818     vrf-table auto
 819
 820 auto eno1
 821 iface eno1 inet manual
 822         mtu 1550
 823
 824 auto vmbr0
 825 iface vmbr0 inet static
 826          address 192.168.0.1
 827          netmask  255.255.255.0
 828          bridge_ports eno1
 829          bridge_stp off
 830          bridge_fd 0
 831
 832 auto vxlan2
 833 iface vxlan2 inet manual
 834         vxlan-id 2
 835         vxlan-local-tunnelip 192.168.0.1
 836         bridge-learning off
 837         bridge-arp-nd-suppress on
 838         bridge-unicast-flood off
 839         bridge-multicast-flood off
 840
 841 auto vmbr2
 842 iface vmbr2 inet static
 843         bridge_ports vxlan2
 844         bridge_stp off
 845         bridge_fd 0
 846         address 10.0.2.254
 847         netmask 255.255.255.0
 848         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 849         vrf vrf1
 850         ip-forward on
 851         ip6-forward on
 852         arp-accept on
 853
 854 auto vxlan3
 855 iface vxlan3 inet manual
 856         vxlan-id 3
 857         vxlan-local-tunnelip 192.168.0.1
 858         bridge-learning off
 859         bridge-arp-nd-suppress on
 860         bridge-unicast-flood off
 861         bridge-multicast-flood off
 862
 863 auto vmbr3
 864 iface vmbr3 inet static
 865         bridge_ports vxlan3
 866         bridge_stp off
 867         bridge_fd 0
 868         address 10.0.3.254
 869         netmask 255.255.255.0
 870         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 871         vrf vrf1
 872         ip-forward on
 873         ip6-forward on
 874         arp-accept on
 875
 876 #interconnect vxlan-vfr l3vni
 877 auto vxlan4000
 878 iface vxlan4000 inet manual
 879         vxlan-id 4000
 880         vxlan-local-tunnelip 192.168.0.1
 881         bridge-learning off
 882         bridge-arp-nd-suppress on
 883         bridge-unicast-flood off
 884         bridge-multicast-flood off
 885
 886
 887 auto vmbr4000
 888 iface vmbr4000 inet manual
 889         bridge_ports vxlan4000
 890         bridge_stp off
 891         bridge_fd 0
 892         vrf vrf1
 893 ----
 894
 895 frr.conf
 896
 897 ----
 898 vrf vrf1
 899  vni 4000
 900  exit-vrf
 901 !
 902 router bgp 1234
 903  bgp router-id 192.168.0.1
 904  no bgp default ipv4-unicast
 905  no bgp default ipv6-unicast
 906  coalesce-time 1000
 907  neighbor 192.168.0.2 remote-as 1234
 908  neighbor 192.168.0.3 remote-as 1234
 909  !
 910  address-family l2vpn evpn
 911   neighbor 192.168.0.2 activate
 912   neighbor 192.168.0.3 activate
 913   advertise-all-vni
 914  exit-address-family
 915 !
 916 line vty
 917 !
 918 ----
 919
 920
 921 * node2
 922
 923 ----
 924 auto vrf1
 925 iface vrf1
 926     vrf-table auto
 927
 928 auto eno1
 929 iface eno1 inet manual
 930         mtu 1550
 931
 932 auto vmbr0
 933 iface vmbr0 inet static
 934          address 192.168.0.2
 935          netmask  255.255.255.0
 936          bridge_ports eno1
 937          bridge_stp off
 938          bridge_fd 0
 939
 940 auto vxlan2
 941 iface vxlan2 inet manual
 942         vxlan-id 2
 943         vxlan-local-tunnelip 192.168.0.2
 944         bridge-learning off
 945         bridge-arp-nd-suppress on
 946         bridge-unicast-flood off
 947         bridge-multicast-flood off
 948
 949 auto vmbr2
 950 iface vmbr2 inet static
 951         bridge_ports vxlan2
 952         bridge_stp off
 953         bridge_fd 0
 954         address 10.0.2.254
 955         netmask 255.255.255.0
 956         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 957         vrf vrf1
 958         ip-forward on
 959         ip6-forward on
 960         arp-accept on
 961
 962 auto vxlan3
 963 iface vxlan3 inet manual
 964         vxlan-id 3
 965         vxlan-local-tunnelip 192.168.0.2
 966         bridge-learning off
 967         bridge-arp-nd-suppress on
 968         bridge-unicast-flood off
 969         bridge-multicast-flood off
 970
 971 auto vmbr3
 972 iface vmbr3 inet static
 973         bridge_ports vxlan3
 974         bridge_stp off
 975         bridge_fd 0
 976         address 10.0.3.254
 977         netmask 255.255.255.0
 978         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 979         vrf vrf1
 980         ip-forward on
 981         ip6-forward on
 982         arp-accept on
 983
 984 #interconnect vxlan-vfr l3vni
 985 auto vxlan4000
 986 iface vxlan4000 inet manual
 987         vxlan-id 4000
 988         vxlan-local-tunnelip 192.168.0.2
 989         bridge-learning off
 990         bridge-arp-nd-suppress on
 991         bridge-unicast-flood off
 992         bridge-multicast-flood off
 993
 994
 995 auto vmbr4000
 996 iface vmbr4000 inet manual
 997         bridge_ports vxlan4000
 998         bridge_stp off
 999         bridge_fd 0
1000         vrf vrf1
1001 ----
1002
1003
1004 frr.conf
1005
1006 ----
1007 vrf vrf1
1008  vni 4000
1009  exit-vrf
1010 !
1011 router bgp 1234
1012  bgp router-id 192.168.0.2
1013  no bgp default ipv4-unicast
1014  no bgp default ipv6-unicast
1015  coalesce-time 1000
1016  neighbor 192.168.0.1 remote-as 1234
1017  neighbor 192.168.0.3 remote-as 1234
1018  !
1019  address-family l2vpn evpn
1020   neighbor 192.168.0.1 activate
1021   neighbor 192.168.0.3 activate
1022   advertise-all-vni
1023  exit-address-family
1024 !
1025 line vty
1026 !
1027 ----
1028
1029
1030 * node3
1031
1032 ----
1033 auto vrf1
1034 iface vrf1
1035     vrf-table auto
1036
1037 auto eno1
1038 iface eno1 inet manual
1039         mtu 1550
1040
1041 auto vmbr0
1042 iface vmbr0 inet static
1043          address 192.168.0.3
1044          netmask  255.255.255.0
1045          bridge_ports eno1
1046          bridge_stp off
1047          bridge_fd 0
1048
1049 auto vxlan2
1050 iface vxlan2 inet manual
1051         vxlan-id 2
1052         vxlan-local-tunnelip 192.168.0.3
1053         bridge-learning off
1054         bridge-arp-nd-suppress on
1055         bridge-unicast-flood off
1056         bridge-multicast-flood off
1057
1058 auto vmbr2
1059 iface vmbr2 inet static
1060         bridge_ports vxlan2
1061         bridge_stp off
1062         bridge_fd 0
1063         address 10.0.2.254
1064         netmask 255.255.255.0
1065         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1066         vrf vrf1
1067         ip-forward on
1068         ip6-forward on
1069         arp-accept on
1070
1071 auto vxlan3
1072 iface vxlan3 inet manual
1073         vxlan-id 3
1074         vxlan-local-tunnelip 192.168.0.3
1075         bridge-learning off
1076         bridge-arp-nd-suppress on
1077         bridge-unicast-flood off
1078         bridge-multicast-flood off
1079
1080 auto vmbr3
1081 iface vmbr3 inet static
1082         bridge_ports vxlan3
1083         bridge_stp off
1084         bridge_fd 0
1085         address 10.0.3.254
1086         netmask 255.255.255.0
1087         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1088         vrf vrf1
1089         ip-forward on
1090         ip6-forward on
1091         arp-accept on
1092
1093 #interconnect vxlan-vfr l3vni
1094 auto vxlan4000
1095 iface vxlan4000 inet manual
1096         vxlan-id 4000
1097         vxlan-local-tunnelip 192.168.0.3
1098         bridge-learning off
1099         bridge-arp-nd-suppress on
1100         bridge-unicast-flood off
1101         bridge-multicast-flood off
1102
1103
1104 auto vmbr4000
1105 iface vmbr4000 inet manual
1106         bridge_ports vxlan4000
1107         bridge_stp off
1108         bridge_fd 0
1109         vrf vrf1
1110 ----
1111
1112
1113 frr.conf
1114
1115 ----
1116 vrf vrf1
1117  vni 4000
1118  exit-vrf
1119 !
1120 router bgp 1234
1121  bgp router-id 192.168.0.3
1122  no bgp default ipv4-unicast
1123  no bgp default ipv6-unicast
1124  coalesce-time 1000
1125  neighbor 192.168.0.1 remote-as 1234
1126  neighbor 192.168.0.2 remote-as 1234
1127  !
1128  address-family l2vpn evpn
1129   neighbor 192.168.0.1 activate
1130   neighbor 192.168.0.2 activate
1131   advertise-all-vni
1132  exit-address-family
1133 !
1134 line vty
1135 !
1136 ----
1137
1138 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1139 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1140 Routing to outside need the symmetric model.
1141
1142 1 gateway node
1143 ^^^^^^^^^^^^^^
1144 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1145 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1146
1147
1148 *node1
1149
1150 ----
1151 auto vrf1
1152 iface vrf1
1153     vrf-table auto
1154
1155 auto eno1
1156 iface eno1 inet manual
1157         mtu 1550
1158
1159 auto vmbr0
1160 iface vmbr0 inet static
1161          address 192.168.0.1
1162          netmask  255.255.255.0
1163          gateway 192.168.0.254
1164          bridge_ports eno1
1165          bridge_stp off
1166          bridge_fd 0
1167          ip-forward on
1168          ip6-forward on
1169
1170 auto vxlan2
1171 iface vxlan2 inet manual
1172         vxlan-id 2
1173         vxlan-local-tunnelip 192.168.0.1
1174         bridge-learning off
1175         bridge-arp-nd-suppress on
1176         bridge-unicast-flood off
1177         bridge-multicast-flood off
1178
1179 auto vmbr2
1180 iface vmbr2 inet static
1181         bridge_ports vxlan2
1182         bridge_stp off
1183         bridge_fd 0
1184         address 10.0.2.254
1185         netmask 255.255.255.0
1186         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1187         vrf vrf1
1188         ip-forward on
1189         ip6-forward on
1190         arp-accept on
1191
1192 auto vxlan3
1193 iface vxlan3 inet manual
1194         vxlan-id 3
1195         vxlan-local-tunnelip 192.168.0.1
1196         bridge-learning off
1197         bridge-arp-nd-suppress on
1198         bridge-unicast-flood off
1199         bridge-multicast-flood off
1200
1201 auto vmbr3
1202 iface vmbr3 inet static
1203         bridge_ports vxlan3
1204         bridge_stp off
1205         bridge_fd 0
1206         address 10.0.3.254
1207         netmask 255.255.255.0
1208         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1209         vrf vrf1
1210         ip-forward on
1211         ip6-forward on
1212         arp-accept on
1213
1214 #interconnect vxlan-vfr l3vni
1215 auto vxlan4000
1216 iface vxlan4000 inet manual
1217         vxlan-id 4000
1218         vxlan-local-tunnelip 192.168.0.1
1219         bridge-learning off
1220         bridge-arp-nd-suppress on
1221         bridge-unicast-flood off
1222         bridge-multicast-flood off
1223
1224 auto vmbr4000
1225 iface vmbr4000 inet manual
1226         bridge_ports vxlan4000
1227         bridge_stp off
1228         bridge_fd 0
1229         vrf vrf1
1230 ----
1231
1232
1233 frr.conf
1234
1235 ----
1236 vrf vrf1
1237  vni 4000
1238  exit-vrf
1239 !
1240 router bgp 1234
1241  bgp router-id 192.168.0.1
1242  no bgp default ipv4-unicast
1243  no bgp default ipv6-unicast
1244  coalesce-time 1000
1245  neighbor 192.168.0.2 remote-as 1234
1246  neighbor 192.168.0.3 remote-as 1234
1247  !
1248  address-family ipv4 unicast
1249   import vrf vrf1
1250  exit-address-family
1251  !
1252  address-family ipv6 unicast
1253   import vrf vrf1
1254  exit-address-family
1255  !
1256  address-family l2vpn evpn
1257   neighbor 192.168.0.2 activate
1258   neighbor 192.168.0.3 activate
1259   advertise-all-vni
1260  exit-address-family
1261 !
1262 router bgp 1234 vrf vrf1
1263 !
1264  address-family ipv4 unicast
1265   redistribute connected
1266  exit-address-family
1267  !
1268  address-family ipv6 unicast
1269   redistribute connected
1270  exit-address-family
1271  !
1272  address-family l2vpn evpn
1273   default-originate ipv4
1274   default-originate ipv6
1275  exit-address-family
1276 !
1277 line vty
1278 !
1279 ----
1280
1281
1282 * node2
1283
1284 ----
1285 auto vrf1
1286 iface vrf1
1287     vrf-table auto
1288
1289 auto eno1
1290 iface eno1 inet manual
1291         mtu 1550
1292
1293 auto vmbr0
1294 iface vmbr0 inet static
1295          address 192.168.0.2
1296          netmask  255.255.255.0
1297          bridge_ports eno1
1298          bridge_stp off
1299          bridge_fd 0
1300
1301 auto vxlan2
1302 iface vxlan2 inet manual
1303         vxlan-id 2
1304         vxlan-local-tunnelip 192.168.0.2
1305         bridge-learning off
1306         bridge-arp-nd-suppress on
1307         bridge-unicast-flood off
1308         bridge-multicast-flood off
1309
1310 auto vmbr2
1311 iface vmbr2 inet static
1312         bridge_ports vxlan2
1313         bridge_stp off
1314         bridge_fd 0
1315         address 10.0.2.254
1316         netmask 255.255.255.0
1317         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1318         vrf vrf1
1319         ip-forward on
1320         ip6-forward on
1321         arp-accept on
1322
1323 auto vxlan3
1324 iface vxlan3 inet manual
1325         vxlan-id 3
1326         vxlan-local-tunnelip 192.168.0.2
1327         bridge-learning off
1328         bridge-arp-nd-suppress on
1329         bridge-unicast-flood off
1330         bridge-multicast-flood off
1331
1332 auto vmbr3
1333 iface vmbr3 inet static
1334         bridge_ports vxlan3
1335         bridge_stp off
1336         bridge_fd 0
1337         address 10.0.3.254
1338         netmask 255.255.255.0
1339         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1340         vrf vrf1
1341         ip-forward on
1342         ip6-forward on
1343         arp-accept on
1344
1345 #interconnect vxlan-vfr l3vni
1346 auto vxlan4000
1347 iface vxlan4000 inet manual
1348         vxlan-id 4000
1349         vxlan-local-tunnelip 192.168.0.2
1350         bridge-learning off
1351         bridge-arp-nd-suppress on
1352         bridge-unicast-flood off
1353         bridge-multicast-flood off
1354
1355
1356 auto vmbr4000
1357 iface vmbr4000 inet manual
1358         bridge_ports vxlan4000
1359         bridge_stp off
1360         bridge_fd 0
1361         vrf vrf1
1362 ----
1363
1364
1365 frr.conf
1366
1367 ----
1368 vrf vrf1
1369  vni 4000
1370  exit-vrf
1371 !
1372 router bgp 1234
1373  bgp router-id 192.168.0.2
1374  no bgp default ipv4-unicast
1375  no bgp default ipv6-unicast
1376  coalesce-time 1000
1377  neighbor 192.168.0.1 remote-as 1234
1378  neighbor 192.168.0.3 remote-as 1234
1379  !
1380  address-family l2vpn evpn
1381   neighbor 192.168.0.1 activate
1382   neighbor 192.168.0.3 activate
1383   advertise-all-vni
1384  exit-address-family
1385 !
1386 line vty
1387 !
1388 ----
1389
1390
1391 * node3
1392
1393 ----
1394 auto vrf1
1395 iface vrf1
1396     vrf-table auto
1397
1398 auto eno1
1399 iface eno1 inet manual
1400         mtu 1550
1401
1402 auto vmbr0
1403 iface vmbr0 inet static
1404          address 192.168.0.3
1405          netmask  255.255.255.0
1406          bridge_ports eno1
1407          bridge_stp off
1408          bridge_fd 0
1409
1410 auto vxlan2
1411 iface vxlan2 inet manual
1412         vxlan-id 2
1413         vxlan-local-tunnelip 192.168.0.3
1414         bridge-learning off
1415         bridge-arp-nd-suppress on
1416         bridge-unicast-flood off
1417         bridge-multicast-flood off
1418
1419 auto vmbr2
1420 iface vmbr2 inet static
1421         bridge_ports vxlan2
1422         bridge_stp off
1423         bridge_fd 0
1424         address 10.0.2.254
1425         netmask 255.255.255.0
1426         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1427         vrf vrf1
1428         ip-forward on
1429         ip6-forward on
1430         arp-accept on
1431
1432 auto vxlan3
1433 iface vxlan3 inet manual
1434         vxlan-id 3
1435         vxlan-local-tunnelip 192.168.0.3
1436         bridge-learning off
1437         bridge-arp-nd-suppress on
1438         bridge-unicast-flood off
1439         bridge-multicast-flood off
1440
1441 auto vmbr3
1442 iface vmbr3 inet static
1443         bridge_ports vxlan3
1444         bridge_stp off
1445         bridge_fd 0
1446         address 10.0.3.254
1447         netmask 255.255.255.0
1448         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1449         vrf vrf1
1450         ip-forward on
1451         ip6-forward on
1452         arp-accept on
1453
1454 #interconnect vxlan-vfr l3vni
1455 auto vxlan4000
1456 iface vxlan4000 inet manual
1457         vxlan-id 4000
1458         vxlan-local-tunnelip 192.168.0.3
1459         bridge-learning off
1460         bridge-arp-nd-suppress on
1461         bridge-unicast-flood off
1462         bridge-multicast-flood off
1463
1464
1465 auto vmbr4000
1466 iface vmbr4000 inet manual
1467         bridge_ports vxlan4000
1468         bridge_stp off
1469         bridge_fd 0
1470         vrf vrf1
1471 ----
1472
1473
1474 frr.conf
1475
1476 ----
1477 vrf vrf1
1478  vni 4000
1479  exit-vrf
1480 !
1481 router bgp 1234
1482  bgp router-id 192.168.0.3
1483  no bgp default ipv4-unicast
1484  no bgp default ipv6-unicast
1485  coalesce-time 1000
1486  neighbor 192.168.0.1 remote-as 1234
1487  neighbor 192.168.0.2 remote-as 1234
1488  !
1489  address-family l2vpn evpn
1490   neighbor 192.168.0.1 activate
1491   neighbor 192.168.0.2 activate
1492   advertise-all-vni
1493  exit-address-family
1494 !
1495 line vty
1496 !
1497 ----
1498
1499 multiple gateway nodes
1500 ^^^^^^^^^^^^^^^^^^^^^^
1501 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1502 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1503 and announce this default gw in the vrf (default originate)
1504 The external router have ecmp routes to all proxmox nodes.(balancing).
1505 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1506 vxlan the packet to final destination.
1507
1508 *node1
1509
1510 ----
1511 auto vrf1
1512 iface vrf1
1513     vrf-table auto
1514
1515 auto eno1
1516 iface eno1 inet manual
1517         mtu 1550
1518
1519 auto vmbr0
1520 iface vmbr0 inet static
1521          address 192.168.0.1
1522          netmask  255.255.255.0
1523          gateway 192.168.0.254
1524          bridge_ports eno1
1525          bridge_stp off
1526          bridge_fd 0
1527          ip-forward on
1528          ip6-forward on
1529
1530 auto vxlan2
1531 iface vxlan2 inet manual
1532         vxlan-id 2
1533         vxlan-local-tunnelip 192.168.0.1
1534         bridge-learning off
1535         bridge-arp-nd-suppress on
1536         bridge-unicast-flood off
1537         bridge-multicast-flood off
1538
1539 auto vmbr2
1540 iface vmbr2 inet static
1541         bridge_ports vxlan2
1542         bridge_stp off
1543         bridge_fd 0
1544         address 10.0.2.254
1545         netmask 255.255.255.0
1546         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1547         vrf vrf1
1548         ip-forward on
1549         ip6-forward on
1550         arp-accept on
1551
1552 auto vxlan3
1553 iface vxlan3 inet manual
1554         vxlan-id 3
1555         vxlan-local-tunnelip 192.168.0.1
1556         bridge-learning off
1557         bridge-arp-nd-suppress on
1558         bridge-unicast-flood off
1559         bridge-multicast-flood off
1560
1561 auto vmbr3
1562 iface vmbr3 inet static
1563         bridge_ports vxlan3
1564         bridge_stp off
1565         bridge_fd 0
1566         address 10.0.3.254
1567         netmask 255.255.255.0
1568         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1569         vrf vrf1
1570         ip-forward on
1571         ip6-forward on
1572         arp-accept on
1573
1574 #interconnect vxlan-vfr l3vni
1575 auto vxlan4000
1576 iface vxlan4000 inet manual
1577         vxlan-id 4000
1578         vxlan-local-tunnelip 192.168.0.1
1579         bridge-learning off
1580         bridge-arp-nd-suppress on
1581         bridge-unicast-flood off
1582         bridge-multicast-flood off
1583
1584 auto vmbr4000
1585 iface vmbr4000 inet manual
1586         bridge_ports vxlan4000
1587         bridge_stp off
1588         bridge_fd 0
1589         vrf vrf1
1590 ----
1591
1592
1593 frr.conf
1594
1595 ----
1596 vrf vrf1
1597  vni 4000
1598  exit-vrf
1599 !
1600 router bgp 1234
1601  bgp router-id 192.168.0.1
1602  no bgp default ipv4-unicast
1603  no bgp default ipv6-unicast
1604  coalesce-time 1000
1605  neighbor 192.168.0.2 remote-as 1234
1606  neighbor 192.168.0.3 remote-as 1234
1607  !
1608  address-family ipv4 unicast
1609   import vrf vrf1
1610  exit-address-family
1611  !
1612  address-family ipv6 unicast
1613   import vrf vrf1
1614  exit-address-family
1615  !
1616  address-family l2vpn evpn
1617   neighbor 192.168.0.2 activate
1618   neighbor 192.168.0.3 activate
1619   advertise-all-vni
1620  exit-address-family
1621 !
1622 router bgp 1234 vrf vrf1
1623 !
1624  address-family ipv4 unicast
1625   redistribute connected
1626  exit-address-family
1627  !
1628  address-family ipv6 unicast
1629   redistribute connected
1630  exit-address-family
1631  !
1632  address-family l2vpn evpn
1633   default-originate ipv4
1634   default-originate ipv6
1635  exit-address-family
1636 !
1637 line vty
1638 !
1639 ----
1640
1641
1642 * node2
1643
1644 ----
1645 auto vrf1
1646 iface vrf1
1647     vrf-table auto
1648
1649 auto eno1
1650 iface eno1 inet manual
1651         mtu 1550
1652
1653 auto vmbr0
1654 iface vmbr0 inet static
1655          address 192.168.0.2
1656          netmask  255.255.255.0
1657          gateway 192.168.0.254
1658          bridge_ports eno1
1659          bridge_stp off
1660          bridge_fd 0
1661          ip-forward on
1662          ip6-forward on
1663
1664 auto vxlan2
1665 iface vxlan2 inet manual
1666         vxlan-id 2
1667         vxlan-local-tunnelip 192.168.0.2
1668         bridge-learning off
1669         bridge-arp-nd-suppress on
1670         bridge-unicast-flood off
1671         bridge-multicast-flood off
1672
1673 auto vmbr2
1674 iface vmbr2 inet static
1675         bridge_ports vxlan2
1676         bridge_stp off
1677         bridge_fd 0
1678         address 10.0.2.254
1679         netmask 255.255.255.0
1680         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1681         vrf vrf1
1682         ip-forward on
1683         ip6-forward on
1684         arp-accept on
1685
1686 auto vxlan3
1687 iface vxlan3 inet manual
1688         vxlan-id 3
1689         vxlan-local-tunnelip 192.168.0.2
1690         bridge-learning off
1691         bridge-arp-nd-suppress on
1692         bridge-unicast-flood off
1693         bridge-multicast-flood off
1694
1695 auto vmbr3
1696 iface vmbr3 inet static
1697         bridge_ports vxlan3
1698         bridge_stp off
1699         bridge_fd 0
1700         address 10.0.3.254
1701         netmask 255.255.255.0
1702         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1703         vrf vrf1
1704         ip-forward on
1705         ip6-forward on
1706         arp-accept on
1707
1708 #interconnect vxlan-vfr l3vni
1709 auto vxlan4000
1710 iface vxlan4000 inet manual
1711         vxlan-id 4000
1712         vxlan-local-tunnelip 192.168.0.2
1713         bridge-learning off
1714         bridge-arp-nd-suppress on
1715         bridge-unicast-flood off
1716         bridge-multicast-flood off
1717
1718
1719 auto vmbr4000
1720 iface vmbr4000 inet manual
1721         bridge_ports vxlan4000
1722         bridge_stp off
1723         bridge_fd 0
1724         vrf vrf1
1725 ----
1726
1727
1728 frr.conf
1729
1730 ----
1731 vrf vrf1
1732  vni 4000
1733  exit-vrf
1734 !
1735 router bgp 1234
1736  bgp router-id 192.168.0.2
1737  no bgp default ipv4-unicast
1738  no bgp default ipv6-unicast
1739  coalesce-time 1000
1740  neighbor 192.168.0.1 remote-as 1234
1741  neighbor 192.168.0.3 remote-as 1234
1742  !
1743  address-family ipv4 unicast
1744   import vrf vrf1
1745  exit-address-family
1746  !
1747  address-family ipv6 unicast
1748   import vrf vrf1
1749  exit-address-family
1750  !
1751  address-family l2vpn evpn
1752   neighbor 192.168.0.1 activate
1753   neighbor 192.168.0.3 activate
1754   advertise-all-vni
1755  exit-address-family
1756 !
1757  address-family ipv4 unicast
1758   redistribute connected
1759  exit-address-family
1760  !
1761  address-family ipv6 unicast
1762   redistribute connected
1763  exit-address-family
1764  !
1765  address-family l2vpn evpn
1766   default-originate ipv4
1767   default-originate ipv6
1768  exit-address-family
1769 !
1770 line vty
1771 !
1772 ----
1773
1774
1775 * node3
1776
1777 ----
1778 auto vrf1
1779 iface vrf1
1780     vrf-table auto
1781
1782 auto eno1
1783 iface eno1 inet manual
1784         mtu 1550
1785
1786 auto vmbr0
1787 iface vmbr0 inet static
1788          address 192.168.0.3
1789          netmask  255.255.255.0
1790          gateway 192.168.0.254
1791          bridge_ports eno1
1792          bridge_stp off
1793          bridge_fd 0
1794          ip-forward on
1795          ip6-forward on
1796
1797 auto vxlan2
1798 iface vxlan2 inet manual
1799         vxlan-id 2
1800         vxlan-local-tunnelip 192.168.0.3
1801         bridge-learning off
1802         bridge-arp-nd-suppress on
1803         bridge-unicast-flood off
1804         bridge-multicast-flood off
1805
1806 auto vmbr2
1807 iface vmbr2 inet static
1808         bridge_ports vxlan2
1809         bridge_stp off
1810         bridge_fd 0
1811         address 10.0.2.254
1812         netmask 255.255.255.0
1813         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1814         vrf vrf1
1815         ip-forward on
1816         ip6-forward on
1817         arp-accept on
1818
1819 auto vxlan3
1820 iface vxlan3 inet manual
1821         vxlan-id 3
1822         vxlan-local-tunnelip 192.168.0.3
1823         bridge-learning off
1824         bridge-arp-nd-suppress on
1825         bridge-unicast-flood off
1826         bridge-multicast-flood off
1827
1828 auto vmbr3
1829 iface vmbr3 inet static
1830         bridge_ports vxlan3
1831         bridge_stp off
1832         bridge_fd 0
1833         address 10.0.3.254
1834         netmask 255.255.255.0
1835         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1836         vrf vrf1
1837         ip-forward on
1838         ip6-forward on
1839         arp-accept on
1840
1841 #interconnect vxlan-vfr l3vni
1842 auto vxlan4000
1843 iface vxlan4000 inet manual
1844         vxlan-id 4000
1845         vxlan-local-tunnelip 192.168.0.3
1846         bridge-learning off
1847         bridge-arp-nd-suppress on
1848         bridge-unicast-flood off
1849         bridge-multicast-flood off
1850
1851
1852 auto vmbr4000
1853 iface vmbr4000 inet manual
1854         bridge_ports vxlan4000
1855         bridge_stp off
1856         bridge_fd 0
1857         vrf vrf1
1858 ----
1859
1860
1861 frr.conf
1862
1863 ----
1864 vrf vrf1
1865  vni 4000
1866  exit-vrf
1867 !
1868 router bgp 1234
1869  bgp router-id 192.168.0.3
1870  no bgp default ipv4-unicast
1871  no bgp default ipv6-unicast
1872  coalesce-time 1000
1873  neighbor 192.168.0.1 remote-as 1234
1874  neighbor 192.168.0.2 remote-as 1234
1875  !
1876  address-family ipv4 unicast
1877   import vrf vrf1
1878  exit-address-family
1879  !
1880  address-family ipv6 unicast
1881   import vrf vrf1
1882  exit-address-family
1883  !
1884  address-family l2vpn evpn
1885   neighbor 192.168.0.1 activate
1886   neighbor 192.168.0.2 activate
1887   advertise-all-vni
1888  exit-address-family
1889 !
1890 router bgp 1234 vrf vrf1
1891 !
1892  address-family ipv4 unicast
1893   redistribute connected
1894  exit-address-family
1895  !
1896  address-family ipv6 unicast
1897   redistribute connected
1898  exit-address-family
1899  !
1900  address-family l2vpn evpn
1901   default-originate ipv4
1902   default-originate ipv6
1903  exit-address-family
1904 !
1905 line vty
1906 !
1907 ----
1908
1909 Note
1910 ^^^^
1911
1912 If your external router doesn't support 'ECMP static routes' to reach multiple
1913 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1914 Virtual Router Redundancy Protocol (VRRP).
1915
1916 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1917 Node1 is the primary with failover to node2 in case of outage.
1918
1919 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1920 #TODO : It should be possible to do it with frr directly with last version.
1921
1922 * node1
1923
1924 ----
1925 auto vmbr0
1926 iface vmbr0 inet static
1927          address 192.168.0.1
1928          netmask  255.255.255.0
1929          gateway 192.168.0.254
1930          bridge_ports eno1
1931          bridge_stp off
1932          bridge_fd 0
1933          vrrp-id 1
1934          vrrp-priority 1
1935          vrrp-virtual-ip 192.168.0.10
1936 ----
1937
1938 * node2
1939
1940 ----
1941 auto vmbr0
1942 iface vmbr0 inet static
1943          address 192.168.0.2
1944          netmask  255.255.255.0
1945          gateway 192.168.0.254
1946          bridge_ports eno1
1947          bridge_stp off
1948          bridge_fd 0
1949          vrrp-id 1
1950          vrrp-priority 2
1951          vrrp-virtual-ip 192.168.0.10
1952 ----
1953
1954
1955 Route Reflectors
1956 ^^^^^^^^^^^^^^^^
1957 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
1958 to avoid that all node peers with each others nodes.
1959 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
1960 single point of failure, a minimum of two servers acting as an RR is highly
1961 recommended for redundancy.
1962
1963 Below is an example of configuration with 'frr', with `rrserver1
1964 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
1965
1966 rrserver1
1967 ----
1968 router bgp 1234
1969   bgp router-id 192.168.0.200
1970   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
1971   bgp log-neighbor-changes
1972   no bgp default ipv4-unicast
1973   no bgp default ipv6-unicast
1974   neighbor fabric peer-group
1975   neighbor fabric remote-as 1234
1976   neighbor fabric capability extended-nexthop
1977   neighbor fabric update-source 192.168.0.200
1978   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
1979   !
1980   address-family l2vpn evpn
1981    neighbor fabric activate
1982    neighbor fabric route-reflector-client
1983    neighbor fabric allowas-in
1984   exit-address-family
1985   !
1986   exit
1987 !
1988 ---
1989
1990 rrserver2
1991 ----
1992 router bgp 1234
1993   bgp router-id 192.168.0.201
1994   bgp cluster-id 1.1.1.1
1995   bgp log-neighbor-changes
1996   no bgp default ipv4-unicast
1997   no bgp default ipv6-unicast
1998   neighbor fabric peer-group
1999   neighbor fabric remote-as 1234
2000   neighbor fabric capability extended-nexthop
2001   neighbor fabric update-source 192.168.0.201
2002   bgp listen range 192.168.0.0/24 peer-group fabric
2003   !
2004   address-family l2vpn evpn
2005    neighbor fabric activate
2006    neighbor fabric route-reflector-client
2007    neighbor fabric allowas-in
2008   exit-address-family
2009   !
2010   exit
2011 !
2012 ---
2013
2014 proxmoxnode(s)
2015 ----
2016 router bgp 1234
2017  bgp router-id 192.168.0.x
2018  no bgp default ipv4-unicast
2019  no bgp default ipv6-unicast
2020  coalesce-time 1000
2021  neighbor 192.168.0.200 remote-as 1234
2022  neighbor 192.168.0.201 remote-as 1234
2023  !
2024  address-family l2vpn evpn
2025   neighbor 192.168.0.200 activate
2026   neighbor 192.168.0.201 activate
2027   advertise-all-vni
2028  exit-address-family
2029 !
2030 ----
2031
2032 #TODO : Documentation with bgp upstream router.