vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  no bgp default ipv6-unicast
 371  coalesce-time 1000
 372  neighbor 192.168.0.2 remote-as 1234
 373  neighbor 192.168.0.3 remote-as 1234
 374  !
 375  address-family l2vpn evpn
 376   neighbor 192.168.0.2 activate
 377   neighbor 192.168.0.3 activate
 378   advertise-all-vni
 379  exit-address-family
 380 !
 381 line vty
 382 !
 383 ----
 384
 385
 386 * node2
 387
 388 ----
 389 auto eno1
 390 iface eno1 inet manual
 391         mtu 1550
 392
 393 auto vmbr0
 394 iface vmbr0 inet static
 395         address 192.168.0.2
 396         netmask 255.255.255.0
 397         bridge_ports eno1
 398         bridge_stp off
 399         bridge_fd 0
 400
 401 auto vxlan2
 402 iface vxlan2 inet manual
 403         vxlan-id 2
 404         vxlan-local-tunnelip 192.168.0.2
 405         bridge-learning off
 406         bridge-arp-nd-suppress on
 407         bridge-unicast-flood off
 408         bridge-multicast-flood off
 409
 410
 411 auto vmbr2
 412 iface vmbr2 inet manual
 413         bridge_ports vxlan2
 414         bridge_stp off
 415         bridge_fd 0
 416
 417 auto vxlan3
 418 iface vxlan3 inet manual
 419         vxlan-id 3
 420         vxlan-local-tunnelip 192.168.0.2
 421         bridge-learning off
 422         bridge-arp-nd-suppress on
 423         bridge-unicast-flood off
 424         bridge-multicast-flood off
 425
 426
 427 auto vmbr3
 428 iface vmbr3 inet manual
 429         bridge_ports vxlan3
 430         bridge_stp off
 431         bridge_fd 0
 432 ----
 433
 434
 435 /etc/frr/frr.conf
 436
 437 ----
 438 router bgp 1234
 439  no bgp default ipv4-unicast
 440  no bgp default ipv6-unicast
 441  coalesce-time 1000
 442  neighbor 192.168.0.1 remote-as 1234
 443  neighbor 192.168.0.3 remote-as 1234
 444  !
 445  address-family l2vpn evpn
 446   neighbor 192.168.0.1 activate
 447   neighbor 192.168.0.3 activate
 448   advertise-all-vni
 449  exit-address-family
 450 !
 451 line vty
 452 !
 453 ----
 454
 455
 456 * node3
 457
 458 ----
 459 auto eno1
 460 iface eno1 inet manual
 461         mtu 1550
 462
 463 auto vmbr0
 464 iface vmbr0 inet static
 465         address 192.168.0.2
 466         netmask 255.255.255.0
 467         bridge_ports eno1
 468         bridge_stp off
 469         bridge_fd 0
 470
 471 auto vxlan2
 472 iface vxlan2 inet manual
 473         vxlan-id 2
 474         vxlan-local-tunnelip 192.168.0.3
 475         bridge-learning off
 476         bridge-arp-nd-suppress on
 477         bridge-unicast-flood off
 478         bridge-multicast-flood off
 479
 480
 481 auto vmbr2
 482 iface vmbr2 inet manual
 483         bridge_ports vxlan2
 484         bridge_stp off
 485         bridge_fd 0
 486
 487 auto vxlan3
 488 iface vxlan3 inet manual
 489         vxlan-id 3
 490         vxlan-local-tunnelip 192.168.0.3
 491         bridge-learning off
 492         bridge-arp-nd-suppress on
 493         bridge-unicast-flood off
 494         bridge-multicast-flood off
 495
 496
 497 auto vmbr3
 498 iface vmbr3 inet manual
 499         bridge_ports vxlan3
 500         bridge_stp off
 501         bridge_fd 0
 502 ----
 503
 504
 505 /etc/frr/frr.conf
 506
 507
 508 ----
 509 router bgp 1234
 510  no bgp default ipv4-unicast
 511  no bgp default ipv6-unicast
 512  coalesce-time 1000
 513  neighbor 192.168.0.1 remote-as 1234
 514  neighbor 192.168.0.2 remote-as 1234
 515  !
 516  address-family l2vpn evpn
 517   neighbor 192.168.0.1 activate
 518   neighbor 192.168.0.2 activate
 519   advertise-all-vni
 520  exit-address-family
 521 !
 522 line vty
 523 !
 524 ----
 525
 526 VXLAN layer3 routing with anycast gateway
 527 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 528
 529 With this need, each vmbr bridge will be the gateway for the vm.
 530 Same vmbr on different node, will have same ip address and same mac address,
 531 to have working vm live migration and no network disruption.
 532
 533 VXLAN layer3 routing only work with FRR and non-aware bridge.
 534 (vlan aware bridge support is buggy currently).
 535
 536 asymmetric model
 537 ^^^^^^^^^^^^^^^^
 538
 539 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 540
 541 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 542 but only bridging on the egress.
 543 This results in bi-directional VXLAN traffic traveling on different VNIs
 544 in each direction (always the destination VNI) across the routed infrastructure.
 545
 546 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 547
 548 * node1
 549
 550 ----
 551 auto eno1
 552 iface eno1 inet manual
 553         mtu 1550
 554
 555 auto vmbr0
 556 iface vmbr0 inet static
 557          address 192.168.0.1
 558          netmask  255.255.255.0
 559          bridge_ports eno1
 560          bridge_stp off
 561          bridge_fd 0
 562
 563 auto vxlan2
 564 iface vxlan2 inet manual
 565         vxlan-id 2
 566         vxlan-local-tunnelip 192.168.0.1
 567         bridge-learning off
 568         bridge-arp-nd-suppress on
 569         bridge-unicast-flood off
 570         bridge-multicast-flood off
 571
 572
 573 auto vmbr2
 574 iface vmbr2 inet static
 575         address 10.0.2.254
 576         netmask 255.255.255.0
 577         hwaddress 44:39:39:FF:40:94
 578         bridge_ports vxlan2
 579         bridge_stp off
 580         bridge_fd 0
 581         ip-forward on
 582         ip6-forward on
 583         arp-accept on
 584
 585 auto vxlan3
 586 iface vxlan3 inet manual
 587         vxlan-id 3
 588         vxlan-local-tunnelip 192.168.0.1
 589         bridge-learning off
 590         bridge-arp-nd-suppress on
 591         bridge-unicast-flood off
 592         bridge-multicast-flood off
 593
 594
 595 auto vmbr3
 596 iface vmbr3 inet static
 597         address 10.0.3.254
 598         netmask 255.255.255.0
 599         hwaddress 44:39:39:FF:40:94
 600         bridge_ports vxlan3
 601         bridge_stp off
 602         bridge_fd 0
 603         ip-forward on
 604         ip6-forward on
 605         arp-accept on
 606 ----
 607
 608
 609 frr.conf
 610
 611 ----
 612 router bgp 1234
 613  bgp router-id 192.168.0.1
 614  no bgp default ipv4-unicast
 615  no bgp default ipv6-unicast
 616  coalesce-time 1000
 617  neighbor 192.168.0.2 remote-as 1234
 618  neighbor 192.168.0.3 remote-as 1234
 619  !
 620  address-family l2vpn evpn
 621   neighbor 192.168.0.2 activate
 622   neighbor 192.168.0.3 activate
 623   advertise-all-vni
 624  exit-address-family
 625 !
 626 line vty
 627 !
 628 ----
 629
 630
 631 * node2
 632
 633 ----
 634 auto eno1
 635 iface eno1 inet manual
 636         mtu 1550
 637
 638 auto vmbr0
 639 iface vmbr0 inet static
 640          address 192.168.0.2
 641          netmask  255.255.255.0
 642          bridge_ports eno1
 643          bridge_stp off
 644          bridge_fd 0
 645
 646 auto vxlan2
 647 iface vxlan2 inet manual
 648         vxlan-id 2
 649         vxlan-local-tunnelip 192.168.0.2
 650         bridge-learning off
 651         bridge-arp-nd-suppress on
 652         bridge-unicast-flood off
 653         bridge-multicast-flood off
 654
 655
 656 auto vmbr2
 657 iface vmbr2 inet static
 658         address 10.0.2.254
 659         netmask 255.255.255.0
 660         hwaddress 44:39:39:FF:40:94
 661         bridge_ports vxlan2
 662         bridge_stp off
 663         bridge_fd 0
 664         ip-forward on
 665         ip6-forward on
 666         arp-accept on
 667
 668
 669 auto vxlan3
 670 iface vxlan3 inet manual
 671         vxlan-id 3
 672         vxlan-local-tunnelip 192.168.0.2
 673         bridge-learning off
 674         bridge-arp-nd-suppress on
 675         bridge-unicast-flood off
 676         bridge-multicast-flood off
 677
 678
 679 auto vmbr3
 680 iface vmbr3 inet static
 681         address 10.0.3.254
 682         netmask 255.255.255.0
 683         hwaddress 44:39:39:FF:40:94
 684         bridge_ports vxlan3
 685         bridge_stp off
 686         bridge_fd 0
 687         ip-forward on
 688         ip6-forward on
 689         arp-accept on
 690 ----
 691
 692
 693 frr.conf
 694
 695 ----
 696 router bgp 1234
 697  bgp router-id 192.168.0.2
 698  no bgp default ipv4-unicast
 699  no bgp default ipv6-unicast
 700  coalesce-time 1000
 701  neighbor 192.168.0.1 remote-as 1234
 702  neighbor 192.168.0.3 remote-as 1234
 703  !
 704  address-family l2vpn evpn
 705   neighbor 192.168.0.1 activate
 706   neighbor 192.168.0.3 activate
 707   advertise-all-vni
 708  exit-address-family
 709 !
 710 line vty
 711 !
 712 ----
 713
 714
 715 * node3
 716
 717 ----
 718 auto eno1
 719 iface eno1 inet manual
 720         mtu 1550
 721
 722 auto vmbr0
 723 iface vmbr0 inet static
 724          address 192.168.0.3
 725          netmask  255.255.255.0
 726          bridge_ports eno1
 727          bridge_stp off
 728          bridge_fd 0
 729
 730 auto vxlan2
 731 iface vxlan2 inet manual
 732         vxlan-id 2
 733         vxlan-local-tunnelip 192.168.0.3
 734         bridge-learning off
 735         bridge-arp-nd-suppress on
 736         bridge-unicast-flood off
 737         bridge-multicast-flood off
 738
 739
 740 auto vmbr2
 741 iface vmbr2 inet static
 742         address 10.0.2.254
 743         netmask 255.255.255.0
 744         hwaddress 44:39:39:FF:40:94
 745         bridge_ports vxlan2
 746         bridge_stp off
 747         bridge_fd 0
 748         ip-forward on
 749         ip6-forward on
 750         arp-accept on
 751
 752 auto vxlan3
 753 iface vxlan3 inet manual
 754         vxlan-id 3
 755         vxlan-local-tunnelip 192.168.0.3
 756         bridge-learning off
 757         bridge-arp-nd-suppress on
 758         bridge-unicast-flood off
 759         bridge-multicast-flood off
 760
 761 auto vmbr3
 762 iface vmbr3 inet static
 763         address 10.0.3.254
 764         netmask 255.255.255.0
 765         hwaddress 44:39:39:FF:40:94
 766         bridge_ports vxlan3
 767         bridge_stp off
 768         bridge_fd 0
 769         ip-forward on
 770         ip6-forward on
 771         arp-accept on
 772 ----
 773
 774
 775 frr.conf
 776
 777 ----
 778 router bgp 1234
 779  bgp router-id 192.168.0.3
 780  no bgp default ipv4-unicast
 781  no bgp default ipv6-unicast
 782  coalesce-time 1000
 783  neighbor 192.168.0.1 remote-as 1234
 784  neighbor 192.168.0.2 remote-as 1234
 785  !
 786  address-family l2vpn evpn
 787   neighbor 192.168.0.1 activate
 788   neighbor 192.168.0.2 activate
 789   advertise-all-vni
 790  exit-address-family
 791 !
 792 line vty
 793 !
 794 ----
 795
 796
 797 symmetric model
 798 ^^^^^^^^^^^^^^^
 799
 800 With this model, you don't need to have all vxlan on all nodes.
 801 This model will also be needed to route traffic to an external router.
 802
 803 The symmetric model routes and bridges on both the ingress and the egress leafs.
 804 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 805 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 806 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 807 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 808
 809 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 810
 811 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 812
 813 * node1
 814
 815 ----
 816 auto vrf1
 817 iface vrf1
 818     vrf-table auto
 819
 820 auto eno1
 821 iface eno1 inet manual
 822         mtu 1550
 823
 824 auto vmbr0
 825 iface vmbr0 inet static
 826          address 192.168.0.1
 827          netmask  255.255.255.0
 828          bridge_ports eno1
 829          bridge_stp off
 830          bridge_fd 0
 831
 832 auto vxlan2
 833 iface vxlan2 inet manual
 834         vxlan-id 2
 835         vxlan-local-tunnelip 192.168.0.1
 836         bridge-learning off
 837         bridge-arp-nd-suppress on
 838         bridge-unicast-flood off
 839         bridge-multicast-flood off
 840
 841 auto vmbr2
 842 iface vmbr2 inet static
 843         bridge_ports vxlan2
 844         bridge_stp off
 845         bridge_fd 0
 846         address 10.0.2.254
 847         netmask 255.255.255.0
 848         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 849         vrf vrf1
 850         ip-forward on
 851         ip6-forward on
 852         arp-accept on
 853
 854 auto vxlan3
 855 iface vxlan3 inet manual
 856         vxlan-id 3
 857         vxlan-local-tunnelip 192.168.0.1
 858         bridge-learning off
 859         bridge-arp-nd-suppress on
 860         bridge-unicast-flood off
 861         bridge-multicast-flood off
 862
 863 auto vmbr3
 864 iface vmbr3 inet static
 865         bridge_ports vxlan3
 866         bridge_stp off
 867         bridge_fd 0
 868         address 10.0.3.254
 869         netmask 255.255.255.0
 870         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 871         vrf vrf1
 872         ip-forward on
 873         ip6-forward on
 874         arp-accept on
 875
 876 #interconnect vxlan-vfr l3vni
 877 auto vxlan4000
 878 iface vxlan4000 inet manual
 879         vxlan-id 4000
 880         vxlan-local-tunnelip 192.168.0.1
 881         bridge-learning off
 882         bridge-arp-nd-suppress on
 883         bridge-unicast-flood off
 884         bridge-multicast-flood off
 885
 886
 887 auto vmbr4000
 888 iface vmbr4000 inet manual
 889         bridge_ports vxlan4000
 890         bridge_stp off
 891         bridge_fd 0
 892         vrf vrf1
 893 ----
 894
 895 frr.conf
 896
 897 ----
 898 vrf vrf1
 899  vni 4000
 900  exit-vrf
 901 !
 902 router bgp 1234
 903  bgp router-id 192.168.0.1
 904  no bgp default ipv4-unicast
 905  no bgp default ipv6-unicast
 906  coalesce-time 1000
 907  neighbor 192.168.0.2 remote-as 1234
 908  neighbor 192.168.0.3 remote-as 1234
 909  !
 910  address-family l2vpn evpn
 911   neighbor 192.168.0.2 activate
 912   neighbor 192.168.0.3 activate
 913   advertise-all-vni
 914  exit-address-family
 915 !
 916 line vty
 917 !
 918 ----
 919
 920
 921 * node2
 922
 923 ----
 924 auto vrf1
 925 iface vrf1
 926     vrf-table auto
 927
 928 auto eno1
 929 iface eno1 inet manual
 930         mtu 1550
 931
 932 auto vmbr0
 933 iface vmbr0 inet static
 934          address 192.168.0.2
 935          netmask  255.255.255.0
 936          bridge_ports eno1
 937          bridge_stp off
 938          bridge_fd 0
 939
 940 auto vxlan2
 941 iface vxlan2 inet manual
 942         vxlan-id 2
 943         vxlan-local-tunnelip 192.168.0.2
 944         bridge-learning off
 945         bridge-arp-nd-suppress on
 946         bridge-unicast-flood off
 947         bridge-multicast-flood off
 948
 949 auto vmbr2
 950 iface vmbr2 inet static
 951         bridge_ports vxlan2
 952         bridge_stp off
 953         bridge_fd 0
 954         address 10.0.2.254
 955         netmask 255.255.255.0
 956         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 957         vrf vrf1
 958         ip-forward on
 959         ip6-forward on
 960         arp-accept on
 961
 962 auto vxlan3
 963 iface vxlan3 inet manual
 964         vxlan-id 3
 965         vxlan-local-tunnelip 192.168.0.2
 966         bridge-learning off
 967         bridge-arp-nd-suppress on
 968         bridge-unicast-flood off
 969         bridge-multicast-flood off
 970
 971 auto vmbr3
 972 iface vmbr3 inet static
 973         bridge_ports vxlan3
 974         bridge_stp off
 975         bridge_fd 0
 976         address 10.0.3.254
 977         netmask 255.255.255.0
 978         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 979         vrf vrf1
 980         ip-forward on
 981         ip6-forward on
 982         arp-accept on
 983
 984 #interconnect vxlan-vfr l3vni
 985 auto vxlan4000
 986 iface vxlan4000 inet manual
 987         vxlan-id 4000
 988         vxlan-local-tunnelip 192.168.0.2
 989         bridge-learning off
 990         bridge-arp-nd-suppress on
 991         bridge-unicast-flood off
 992         bridge-multicast-flood off
 993
 994
 995 auto vmbr4000
 996 iface vmbr4000 inet manual
 997         bridge_ports vxlan4000
 998         bridge_stp off
 999         bridge_fd 0
1000         vrf vrf1
1001 ----
1002
1003
1004 frr.conf
1005
1006 ----
1007 vrf vrf1
1008  vni 4000
1009  exit-vrf
1010 !
1011 router bgp 1234
1012  bgp router-id 192.168.0.2
1013  no bgp default ipv4-unicast
1014  no bgp default ipv6-unicast
1015  coalesce-time 1000
1016  neighbor 192.168.0.1 remote-as 1234
1017  neighbor 192.168.0.3 remote-as 1234
1018  !
1019  address-family l2vpn evpn
1020   neighbor 192.168.0.1 activate
1021   neighbor 192.168.0.3 activate
1022   advertise-all-vni
1023  exit-address-family
1024 !
1025 line vty
1026 !
1027 ----
1028
1029
1030 * node3
1031
1032 ----
1033 auto vrf1
1034 iface vrf1
1035     vrf-table auto
1036
1037 auto eno1
1038 iface eno1 inet manual
1039         mtu 1550
1040
1041 auto vmbr0
1042 iface vmbr0 inet static
1043          address 192.168.0.3
1044          netmask  255.255.255.0
1045          bridge_ports eno1
1046          bridge_stp off
1047          bridge_fd 0
1048
1049 auto vxlan2
1050 iface vxlan2 inet manual
1051         vxlan-id 2
1052         vxlan-local-tunnelip 192.168.0.3
1053         bridge-learning off
1054         bridge-arp-nd-suppress on
1055         bridge-unicast-flood off
1056         bridge-multicast-flood off
1057
1058 auto vmbr2
1059 iface vmbr2 inet static
1060         bridge_ports vxlan2
1061         bridge_stp off
1062         bridge_fd 0
1063         address 10.0.2.254
1064         netmask 255.255.255.0
1065         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1066         vrf vrf1
1067         ip-forward on
1068         ip6-forward on
1069         arp-accept on
1070
1071 auto vxlan3
1072 iface vxlan3 inet manual
1073         vxlan-id 3
1074         vxlan-local-tunnelip 192.168.0.3
1075         bridge-learning off
1076         bridge-arp-nd-suppress on
1077         bridge-unicast-flood off
1078         bridge-multicast-flood off
1079
1080 auto vmbr3
1081 iface vmbr3 inet static
1082         bridge_ports vxlan3
1083         bridge_stp off
1084         bridge_fd 0
1085         address 10.0.3.254
1086         netmask 255.255.255.0
1087         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1088         vrf vrf1
1089         ip-forward on
1090         ip6-forward on
1091         arp-accept on
1092
1093 #interconnect vxlan-vfr l3vni
1094 auto vxlan4000
1095 iface vxlan4000 inet manual
1096         vxlan-id 4000
1097         vxlan-local-tunnelip 192.168.0.3
1098         bridge-learning off
1099         bridge-arp-nd-suppress on
1100         bridge-unicast-flood off
1101         bridge-multicast-flood off
1102
1103
1104 auto vmbr4000
1105 iface vmbr4000 inet manual
1106         bridge_ports vxlan4000
1107         bridge_stp off
1108         bridge_fd 0
1109         vrf vrf1
1110 ----
1111
1112
1113 frr.conf
1114
1115 ----
1116 vrf vrf1
1117  vni 4000
1118  exit-vrf
1119 !
1120 router bgp 1234
1121  bgp router-id 192.168.0.3
1122  no bgp default ipv4-unicast
1123  no bgp default ipv6-unicast
1124  coalesce-time 1000
1125  neighbor 192.168.0.1 remote-as 1234
1126  neighbor 192.168.0.2 remote-as 1234
1127  !
1128  address-family l2vpn evpn
1129   neighbor 192.168.0.1 activate
1130   neighbor 192.168.0.2 activate
1131   advertise-all-vni
1132  exit-address-family
1133 !
1134 line vty
1135 !
1136 ----
1137
1138 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1139 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1140 Routing to outside need the symmetric model.
1141
1142 1 gateway node
1143 ^^^^^^^^^^^^^^
1144 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1145 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1146
1147
1148 *node1
1149
1150 ----
1151 auto vrf1
1152 iface vrf1
1153     vrf-table auto
1154
1155 auto eno1
1156 iface eno1 inet manual
1157         mtu 1550
1158
1159 auto vmbr0
1160 iface vmbr0 inet static
1161          address 192.168.0.1
1162          netmask  255.255.255.0
1163          gateway 192.168.0.254
1164          bridge_ports eno1
1165          bridge_stp off
1166          bridge_fd 0
1167          ip-forward on
1168          ip6-forward on
1169
1170 auto vxlan2
1171 iface vxlan2 inet manual
1172         vxlan-id 2
1173         vxlan-local-tunnelip 192.168.0.1
1174         bridge-learning off
1175         bridge-arp-nd-suppress on
1176         bridge-unicast-flood off
1177         bridge-multicast-flood off
1178
1179 auto vmbr2
1180 iface vmbr2 inet static
1181         bridge_ports vxlan2
1182         bridge_stp off
1183         bridge_fd 0
1184         address 10.0.2.254
1185         netmask 255.255.255.0
1186         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1187         vrf vrf1
1188         ip-forward on
1189         ip6-forward on
1190         arp-accept on
1191
1192 auto vxlan3
1193 iface vxlan3 inet manual
1194         vxlan-id 3
1195         vxlan-local-tunnelip 192.168.0.1
1196         bridge-learning off
1197         bridge-arp-nd-suppress on
1198         bridge-unicast-flood off
1199         bridge-multicast-flood off
1200
1201 auto vmbr3
1202 iface vmbr3 inet static
1203         bridge_ports vxlan3
1204         bridge_stp off
1205         bridge_fd 0
1206         address 10.0.3.254
1207         netmask 255.255.255.0
1208         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1209         vrf vrf1
1210         ip-forward on
1211         ip6-forward on
1212         arp-accept on
1213
1214 #interconnect vxlan-vfr l3vni
1215 auto vxlan4000
1216 iface vxlan4000 inet manual
1217         vxlan-id 4000
1218         vxlan-local-tunnelip 192.168.0.1
1219         bridge-learning off
1220         bridge-arp-nd-suppress on
1221         bridge-unicast-flood off
1222         bridge-multicast-flood off
1223
1224 auto vmbr4000
1225 iface vmbr4000 inet manual
1226         bridge_ports vxlan4000
1227         bridge_stp off
1228         bridge_fd 0
1229         vrf vrf1
1230 ----
1231
1232
1233 frr.conf
1234
1235 ----
1236 vrf vrf1
1237  vni 4000
1238  exit-vrf
1239 !
1240 router bgp 1234
1241  bgp router-id 192.168.0.1
1242  no bgp default ipv4-unicast
1243  no bgp default ipv6-unicast
1244  coalesce-time 1000
1245  neighbor 192.168.0.2 remote-as 1234
1246  neighbor 192.168.0.3 remote-as 1234
1247  !
1248  address-family ipv4 unicast
1249   import vrf vrf1
1250  exit-address-family
1251  !
1252  address-family l2vpn evpn
1253   neighbor 192.168.0.2 activate
1254   neighbor 192.168.0.3 activate
1255   advertise-all-vni
1256  exit-address-family
1257 !
1258 router bgp 1234 vrf vrf1
1259 !
1260  address-family ipv4 unicast
1261   redistribute connected
1262  exit-address-family
1263  !
1264  address-family l2vpn evpn
1265   default-originate ipv4
1266  exit-address-family
1267 !
1268 line vty
1269 !
1270 ----
1271
1272
1273 * node2
1274
1275 ----
1276 auto vrf1
1277 iface vrf1
1278     vrf-table auto
1279
1280 auto eno1
1281 iface eno1 inet manual
1282         mtu 1550
1283
1284 auto vmbr0
1285 iface vmbr0 inet static
1286          address 192.168.0.2
1287          netmask  255.255.255.0
1288          bridge_ports eno1
1289          bridge_stp off
1290          bridge_fd 0
1291
1292 auto vxlan2
1293 iface vxlan2 inet manual
1294         vxlan-id 2
1295         vxlan-local-tunnelip 192.168.0.2
1296         bridge-learning off
1297         bridge-arp-nd-suppress on
1298         bridge-unicast-flood off
1299         bridge-multicast-flood off
1300
1301 auto vmbr2
1302 iface vmbr2 inet static
1303         bridge_ports vxlan2
1304         bridge_stp off
1305         bridge_fd 0
1306         address 10.0.2.254
1307         netmask 255.255.255.0
1308         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1309         vrf vrf1
1310         ip-forward on
1311         ip6-forward on
1312         arp-accept on
1313
1314 auto vxlan3
1315 iface vxlan3 inet manual
1316         vxlan-id 3
1317         vxlan-local-tunnelip 192.168.0.2
1318         bridge-learning off
1319         bridge-arp-nd-suppress on
1320         bridge-unicast-flood off
1321         bridge-multicast-flood off
1322
1323 auto vmbr3
1324 iface vmbr3 inet static
1325         bridge_ports vxlan3
1326         bridge_stp off
1327         bridge_fd 0
1328         address 10.0.3.254
1329         netmask 255.255.255.0
1330         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1331         vrf vrf1
1332         ip-forward on
1333         ip6-forward on
1334         arp-accept on
1335
1336 #interconnect vxlan-vfr l3vni
1337 auto vxlan4000
1338 iface vxlan4000 inet manual
1339         vxlan-id 4000
1340         vxlan-local-tunnelip 192.168.0.2
1341         bridge-learning off
1342         bridge-arp-nd-suppress on
1343         bridge-unicast-flood off
1344         bridge-multicast-flood off
1345
1346
1347 auto vmbr4000
1348 iface vmbr4000 inet manual
1349         bridge_ports vxlan4000
1350         bridge_stp off
1351         bridge_fd 0
1352         vrf vrf1
1353 ----
1354
1355
1356 frr.conf
1357
1358 ----
1359 vrf vrf1
1360  vni 4000
1361  exit-vrf
1362 !
1363 router bgp 1234
1364  bgp router-id 192.168.0.2
1365  no bgp default ipv4-unicast
1366  no bgp default ipv6-unicast
1367  coalesce-time 1000
1368  neighbor 192.168.0.1 remote-as 1234
1369  neighbor 192.168.0.3 remote-as 1234
1370  !
1371  address-family l2vpn evpn
1372   neighbor 192.168.0.1 activate
1373   neighbor 192.168.0.3 activate
1374   advertise-all-vni
1375  exit-address-family
1376 !
1377 line vty
1378 !
1379 ----
1380
1381
1382 * node3
1383
1384 ----
1385 auto vrf1
1386 iface vrf1
1387     vrf-table auto
1388
1389 auto eno1
1390 iface eno1 inet manual
1391         mtu 1550
1392
1393 auto vmbr0
1394 iface vmbr0 inet static
1395          address 192.168.0.3
1396          netmask  255.255.255.0
1397          bridge_ports eno1
1398          bridge_stp off
1399          bridge_fd 0
1400
1401 auto vxlan2
1402 iface vxlan2 inet manual
1403         vxlan-id 2
1404         vxlan-local-tunnelip 192.168.0.3
1405         bridge-learning off
1406         bridge-arp-nd-suppress on
1407         bridge-unicast-flood off
1408         bridge-multicast-flood off
1409
1410 auto vmbr2
1411 iface vmbr2 inet static
1412         bridge_ports vxlan2
1413         bridge_stp off
1414         bridge_fd 0
1415         address 10.0.2.254
1416         netmask 255.255.255.0
1417         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1418         vrf vrf1
1419         ip-forward on
1420         ip6-forward on
1421         arp-accept on
1422
1423 auto vxlan3
1424 iface vxlan3 inet manual
1425         vxlan-id 3
1426         vxlan-local-tunnelip 192.168.0.3
1427         bridge-learning off
1428         bridge-arp-nd-suppress on
1429         bridge-unicast-flood off
1430         bridge-multicast-flood off
1431
1432 auto vmbr3
1433 iface vmbr3 inet static
1434         bridge_ports vxlan3
1435         bridge_stp off
1436         bridge_fd 0
1437         address 10.0.3.254
1438         netmask 255.255.255.0
1439         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1440         vrf vrf1
1441         ip-forward on
1442         ip6-forward on
1443         arp-accept on
1444
1445 #interconnect vxlan-vfr l3vni
1446 auto vxlan4000
1447 iface vxlan4000 inet manual
1448         vxlan-id 4000
1449         vxlan-local-tunnelip 192.168.0.3
1450         bridge-learning off
1451         bridge-arp-nd-suppress on
1452         bridge-unicast-flood off
1453         bridge-multicast-flood off
1454
1455
1456 auto vmbr4000
1457 iface vmbr4000 inet manual
1458         bridge_ports vxlan4000
1459         bridge_stp off
1460         bridge_fd 0
1461         vrf vrf1
1462 ----
1463
1464
1465 frr.conf
1466
1467 ----
1468 vrf vrf1
1469  vni 4000
1470  exit-vrf
1471 !
1472 router bgp 1234
1473  bgp router-id 192.168.0.3
1474  no bgp default ipv4-unicast
1475  no bgp default ipv6-unicast
1476  coalesce-time 1000
1477  neighbor 192.168.0.1 remote-as 1234
1478  neighbor 192.168.0.2 remote-as 1234
1479  !
1480  address-family l2vpn evpn
1481   neighbor 192.168.0.1 activate
1482   neighbor 192.168.0.2 activate
1483   advertise-all-vni
1484  exit-address-family
1485 !
1486 line vty
1487 !
1488 ----
1489
1490 multiple gateway nodes
1491 ^^^^^^^^^^^^^^^^^^^^^^
1492 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1493 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1494 and announce this default gw in the vrf (default originate)
1495 The external router have ecmp routes to all proxmox nodes.(balancing).
1496 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1497 vxlan the packet to final destination.
1498
1499 *node1
1500
1501 ----
1502 auto vrf1
1503 iface vrf1
1504     vrf-table auto
1505
1506 auto eno1
1507 iface eno1 inet manual
1508         mtu 1550
1509
1510 auto vmbr0
1511 iface vmbr0 inet static
1512          address 192.168.0.1
1513          netmask  255.255.255.0
1514          gateway 192.168.0.254
1515          bridge_ports eno1
1516          bridge_stp off
1517          bridge_fd 0
1518          ip-forward on
1519          ip6-forward on
1520
1521 auto vxlan2
1522 iface vxlan2 inet manual
1523         vxlan-id 2
1524         vxlan-local-tunnelip 192.168.0.1
1525         bridge-learning off
1526         bridge-arp-nd-suppress on
1527         bridge-unicast-flood off
1528         bridge-multicast-flood off
1529
1530 auto vmbr2
1531 iface vmbr2 inet static
1532         bridge_ports vxlan2
1533         bridge_stp off
1534         bridge_fd 0
1535         address 10.0.2.254
1536         netmask 255.255.255.0
1537         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1538         vrf vrf1
1539         ip-forward on
1540         ip6-forward on
1541         arp-accept on
1542
1543 auto vxlan3
1544 iface vxlan3 inet manual
1545         vxlan-id 3
1546         vxlan-local-tunnelip 192.168.0.1
1547         bridge-learning off
1548         bridge-arp-nd-suppress on
1549         bridge-unicast-flood off
1550         bridge-multicast-flood off
1551
1552 auto vmbr3
1553 iface vmbr3 inet static
1554         bridge_ports vxlan3
1555         bridge_stp off
1556         bridge_fd 0
1557         address 10.0.3.254
1558         netmask 255.255.255.0
1559         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1560         vrf vrf1
1561         ip-forward on
1562         ip6-forward on
1563         arp-accept on
1564
1565 #interconnect vxlan-vfr l3vni
1566 auto vxlan4000
1567 iface vxlan4000 inet manual
1568         vxlan-id 4000
1569         vxlan-local-tunnelip 192.168.0.1
1570         bridge-learning off
1571         bridge-arp-nd-suppress on
1572         bridge-unicast-flood off
1573         bridge-multicast-flood off
1574
1575 auto vmbr4000
1576 iface vmbr4000 inet manual
1577         bridge_ports vxlan4000
1578         bridge_stp off
1579         bridge_fd 0
1580         vrf vrf1
1581 ----
1582
1583
1584 frr.conf
1585
1586 ----
1587 vrf vrf1
1588  vni 4000
1589  exit-vrf
1590 !
1591 router bgp 1234
1592  bgp router-id 192.168.0.1
1593  no bgp default ipv4-unicast
1594  no bgp default ipv6-unicast
1595  coalesce-time 1000
1596  neighbor 192.168.0.2 remote-as 1234
1597  neighbor 192.168.0.3 remote-as 1234
1598  !
1599  address-family ipv4 unicast
1600   import vrf vrf1
1601  exit-address-family
1602  !
1603  address-family l2vpn evpn
1604   neighbor 192.168.0.2 activate
1605   neighbor 192.168.0.3 activate
1606   advertise-all-vni
1607  exit-address-family
1608 !
1609 router bgp 1234 vrf vrf1
1610 !
1611  address-family ipv4 unicast
1612   redistribute connected
1613  exit-address-family
1614  !
1615  address-family l2vpn evpn
1616   default-originate ipv4
1617  exit-address-family
1618 !
1619 line vty
1620 !
1621 ----
1622
1623
1624 * node2
1625
1626 ----
1627 auto vrf1
1628 iface vrf1
1629     vrf-table auto
1630
1631 auto eno1
1632 iface eno1 inet manual
1633         mtu 1550
1634
1635 auto vmbr0
1636 iface vmbr0 inet static
1637          address 192.168.0.2
1638          netmask  255.255.255.0
1639          gateway 192.168.0.254
1640          bridge_ports eno1
1641          bridge_stp off
1642          bridge_fd 0
1643          ip-forward on
1644          ip6-forward on
1645
1646 auto vxlan2
1647 iface vxlan2 inet manual
1648         vxlan-id 2
1649         vxlan-local-tunnelip 192.168.0.2
1650         bridge-learning off
1651         bridge-arp-nd-suppress on
1652         bridge-unicast-flood off
1653         bridge-multicast-flood off
1654
1655 auto vmbr2
1656 iface vmbr2 inet static
1657         bridge_ports vxlan2
1658         bridge_stp off
1659         bridge_fd 0
1660         address 10.0.2.254
1661         netmask 255.255.255.0
1662         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1663         vrf vrf1
1664         ip-forward on
1665         ip6-forward on
1666         arp-accept on
1667
1668 auto vxlan3
1669 iface vxlan3 inet manual
1670         vxlan-id 3
1671         vxlan-local-tunnelip 192.168.0.2
1672         bridge-learning off
1673         bridge-arp-nd-suppress on
1674         bridge-unicast-flood off
1675         bridge-multicast-flood off
1676
1677 auto vmbr3
1678 iface vmbr3 inet static
1679         bridge_ports vxlan3
1680         bridge_stp off
1681         bridge_fd 0
1682         address 10.0.3.254
1683         netmask 255.255.255.0
1684         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1685         vrf vrf1
1686         ip-forward on
1687         ip6-forward on
1688         arp-accept on
1689
1690 #interconnect vxlan-vfr l3vni
1691 auto vxlan4000
1692 iface vxlan4000 inet manual
1693         vxlan-id 4000
1694         vxlan-local-tunnelip 192.168.0.2
1695         bridge-learning off
1696         bridge-arp-nd-suppress on
1697         bridge-unicast-flood off
1698         bridge-multicast-flood off
1699
1700
1701 auto vmbr4000
1702 iface vmbr4000 inet manual
1703         bridge_ports vxlan4000
1704         bridge_stp off
1705         bridge_fd 0
1706         vrf vrf1
1707 ----
1708
1709
1710 frr.conf
1711
1712 ----
1713 vrf vrf1
1714  vni 4000
1715  exit-vrf
1716 !
1717 router bgp 1234
1718  bgp router-id 192.168.0.2
1719  no bgp default ipv4-unicast
1720  no bgp default ipv6-unicast
1721  coalesce-time 1000
1722  neighbor 192.168.0.1 remote-as 1234
1723  neighbor 192.168.0.3 remote-as 1234
1724  !
1725  address-family ipv4 unicast
1726   import vrf vrf1
1727  exit-address-family
1728  !
1729  address-family l2vpn evpn
1730   neighbor 192.168.0.1 activate
1731   neighbor 192.168.0.3 activate
1732   advertise-all-vni
1733  exit-address-family
1734 !
1735  address-family ipv4 unicast
1736   redistribute connected
1737  exit-address-family
1738  !
1739  address-family l2vpn evpn
1740   default-originate ipv4
1741  exit-address-family
1742 !
1743 line vty
1744 !
1745 ----
1746
1747
1748 * node3
1749
1750 ----
1751 auto vrf1
1752 iface vrf1
1753     vrf-table auto
1754
1755 auto eno1
1756 iface eno1 inet manual
1757         mtu 1550
1758
1759 auto vmbr0
1760 iface vmbr0 inet static
1761          address 192.168.0.3
1762          netmask  255.255.255.0
1763          gateway 192.168.0.254
1764          bridge_ports eno1
1765          bridge_stp off
1766          bridge_fd 0
1767          ip-forward on
1768          ip6-forward on
1769
1770 auto vxlan2
1771 iface vxlan2 inet manual
1772         vxlan-id 2
1773         vxlan-local-tunnelip 192.168.0.3
1774         bridge-learning off
1775         bridge-arp-nd-suppress on
1776         bridge-unicast-flood off
1777         bridge-multicast-flood off
1778
1779 auto vmbr2
1780 iface vmbr2 inet static
1781         bridge_ports vxlan2
1782         bridge_stp off
1783         bridge_fd 0
1784         address 10.0.2.254
1785         netmask 255.255.255.0
1786         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1787         vrf vrf1
1788         ip-forward on
1789         ip6-forward on
1790         arp-accept on
1791
1792 auto vxlan3
1793 iface vxlan3 inet manual
1794         vxlan-id 3
1795         vxlan-local-tunnelip 192.168.0.3
1796         bridge-learning off
1797         bridge-arp-nd-suppress on
1798         bridge-unicast-flood off
1799         bridge-multicast-flood off
1800
1801 auto vmbr3
1802 iface vmbr3 inet static
1803         bridge_ports vxlan3
1804         bridge_stp off
1805         bridge_fd 0
1806         address 10.0.3.254
1807         netmask 255.255.255.0
1808         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1809         vrf vrf1
1810         ip-forward on
1811         ip6-forward on
1812         arp-accept on
1813
1814 #interconnect vxlan-vfr l3vni
1815 auto vxlan4000
1816 iface vxlan4000 inet manual
1817         vxlan-id 4000
1818         vxlan-local-tunnelip 192.168.0.3
1819         bridge-learning off
1820         bridge-arp-nd-suppress on
1821         bridge-unicast-flood off
1822         bridge-multicast-flood off
1823
1824
1825 auto vmbr4000
1826 iface vmbr4000 inet manual
1827         bridge_ports vxlan4000
1828         bridge_stp off
1829         bridge_fd 0
1830         vrf vrf1
1831 ----
1832
1833
1834 frr.conf
1835
1836 ----
1837 vrf vrf1
1838  vni 4000
1839  exit-vrf
1840 !
1841 router bgp 1234
1842  bgp router-id 192.168.0.3
1843  no bgp default ipv4-unicast
1844  no bgp default ipv6-unicast
1845  coalesce-time 1000
1846  neighbor 192.168.0.1 remote-as 1234
1847  neighbor 192.168.0.2 remote-as 1234
1848  !
1849  address-family ipv4 unicast
1850   import vrf vrf1
1851  exit-address-family
1852  !
1853  address-family l2vpn evpn
1854   neighbor 192.168.0.1 activate
1855   neighbor 192.168.0.2 activate
1856   advertise-all-vni
1857  exit-address-family
1858 !
1859 router bgp 1234 vrf vrf1
1860 !
1861  address-family ipv4 unicast
1862   redistribute connected
1863  exit-address-family
1864  !
1865  address-family l2vpn evpn
1866   default-originate ipv4
1867  exit-address-family
1868 !
1869 line vty
1870 !
1871 ----
1872
1873 Note
1874 ^^^^
1875
1876 If your external router doesn't support 'ECMP static routes' to reach multiple
1877 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1878 Virtual Router Redundancy Protocol (VRRP).
1879
1880 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1881 Node1 is the primary with failover to node2 in case of outage.
1882
1883 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1884 #TODO : It should be possible to do it with frr directly with last version.
1885
1886 * node1
1887
1888 ----
1889 auto vmbr0
1890 iface vmbr0 inet static
1891          address 192.168.0.1
1892          netmask  255.255.255.0
1893          gateway 192.168.0.254
1894          bridge_ports eno1
1895          bridge_stp off
1896          bridge_fd 0
1897          vrrp-id 1
1898          vrrp-priority 1
1899          vrrp-virtual-ip 192.168.0.10
1900 ----
1901
1902 * node2
1903
1904 ----
1905 auto vmbr0
1906 iface vmbr0 inet static
1907          address 192.168.0.2
1908          netmask  255.255.255.0
1909          gateway 192.168.0.254
1910          bridge_ports eno1
1911          bridge_stp off
1912          bridge_fd 0
1913          vrrp-id 1
1914          vrrp-priority 2
1915          vrrp-virtual-ip 192.168.0.10
1916 ----
1917
1918
1919 Route Reflectors
1920 ^^^^^^^^^^^^^^^^
1921 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
1922 to avoid that all node peers with each others nodes.
1923 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
1924 single point of failure, a minimum of two servers acting as an RR is highly
1925 recommended for redundancy.
1926
1927 Below is an example of configuration with 'frr', with `rrserver1
1928 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
1929
1930 rrserver1
1931 ----
1932 router bgp 1234
1933   bgp router-id 192.168.0.200
1934   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
1935   bgp log-neighbor-changes
1936   no bgp default ipv4-unicast
1937   no bgp default ipv6-unicast
1938   neighbor fabric peer-group
1939   neighbor fabric remote-as 1234
1940   neighbor fabric capability extended-nexthop
1941   neighbor fabric update-source 192.168.0.200
1942   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
1943   !
1944   address-family l2vpn evpn
1945    neighbor fabric activate
1946    neighbor fabric route-reflector-client
1947    neighbor fabric allowas-in
1948   exit-address-family
1949   !
1950   exit
1951 !
1952 ---
1953
1954 rrserver2
1955 ----
1956 router bgp 1234
1957   bgp router-id 192.168.0.201
1958   bgp cluster-id 1.1.1.1
1959   bgp log-neighbor-changes
1960   no bgp default ipv4-unicast
1961   no bgp default ipv6-unicast
1962   neighbor fabric peer-group
1963   neighbor fabric remote-as 1234
1964   neighbor fabric capability extended-nexthop
1965   neighbor fabric update-source 192.168.0.201
1966   bgp listen range 192.168.0.0/24 peer-group fabric
1967   !
1968   address-family l2vpn evpn
1969    neighbor fabric activate
1970    neighbor fabric route-reflector-client
1971    neighbor fabric allowas-in
1972   exit-address-family
1973   !
1974   exit
1975 !
1976 ---
1977
1978 proxmoxnode(s)
1979 ----
1980 router bgp 1234
1981  bgp router-id 192.168.0.x
1982  no bgp default ipv4-unicast
1983  no bgp default ipv6-unicast
1984  coalesce-time 1000
1985  neighbor 192.168.0.200 remote-as 1234
1986  neighbor 192.168.0.201 remote-as 1234
1987  !
1988  address-family l2vpn evpn
1989   neighbor 192.168.0.200 activate
1990   neighbor 192.168.0.201 activate
1991   advertise-all-vni
1992  exit-address-family
1993 !
1994 ----
1995
1996 #TODO : Documentation with bgp upstream router.