vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-id 2
  55         vxlan-svcnodeip 225.20.1.1
  56         vxlan-physdev eno1
  57
  58 auto vmbr2
  59 iface vmbr2 inet manual
  60         bridge_ports vxlan2
  61         bridge_stp off
  62         bridge_fd 0
  63
  64 auto vxlan3
  65 iface vxlan3 inet manual
  66         vxlan-id 3
  67         vxlan-svcnodeip 225.20.1.1
  68         vxlan-physdev eno1
  69
  70 auto vmbr3
  71 iface vmbr3 inet manual
  72         bridge_ports vxlan3
  73         bridge_stp off
  74         bridge_fd 0
  75 ----
  76
  77
  78 * node2
  79
  80 ----
  81 auto eno1
  82 iface eno1 inet manual
  83         mtu 1550
  84
  85 auto vmbr0
  86 iface vmbr0 inet static
  87         address 192.168.0.2
  88         netmask  255.255.255.0
  89         bridge_ports eno1
  90         bridge_stp off
  91         bridge_fd 0
  92
  93 auto vxlan2
  94 iface vxlan2 inet manual
  95         vxlan-id 2
  96         vxlan-svcnodeip 225.20.1.1
  97         vxlan-physdev eno1
  98
  99 auto vmbr2
 100 iface vmbr2 inet manual
 101         bridge_ports vxlan2
 102         bridge_stp off
 103         bridge_fd 0
 104
 105
 106 auto vxlan3
 107 iface vxlan3 inet manual
 108         vxlan-id 3
 109         vxlan-svcnodeip 225.20.1.1
 110         vxlan-physdev eno1
 111
 112 auto vmbr3
 113 iface vmbr3 inet manual
 114         bridge_ports vxlan3
 115         bridge_stp off
 116         bridge_fd 0
 117 ----
 118
 119
 120 * node3
 121
 122 ----
 123 auto eno1
 124 iface eno1 inet manual
 125         mtu 1550
 126
 127 auto vmbr0
 128 iface vmbr0 inet static
 129         address 192.168.0.3
 130         netmask  255.255.255.0
 131         bridge_ports eno1
 132         bridge_stp off
 133         bridge_fd 0
 134
 135 auto vxlan2
 136 iface vxlan2 inet manual
 137         vxlan-id 2
 138         vxlan-svcnodeip 225.20.1.1
 139         vxlan-physdev eno1
 140
 141 auto vmbr2
 142 iface vmbr2 inet manual
 143         bridge_ports vxlan2
 144         bridge_stp off
 145         bridge_fd 0
 146
 147
 148 auto vxlan3
 149 iface vxlan3 inet manual
 150         vxlan-id 3
 151         vxlan-svcnodeip 225.20.1.1
 152         vxlan-physdev eno1
 153
 154 auto vmbr3
 155 iface vmbr3 inet manual
 156         bridge_ports vxlan3
 157         bridge_stp off
 158         bridge_fd 0
 159 ----
 160
 161
 162 unicast mode
 163 ^^^^^^^^^^^^
 164
 165 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 166 The VXLAN is defined without a remote multicast group.
 167 Instead, all the remote VTEPs are associated with the all-zero address:
 168 a BUM frame will be duplicated to all these destinations.
 169 The VXLAN device will still learn remote addresses automatically using source-address learning.
 170
 171 * node1
 172
 173 ----
 174 auto eno1
 175 iface eno1 inet manual
 176         mtu 1550
 177
 178 auto vmbr0
 179 iface vmbr0 inet static
 180         address 192.168.0.1
 181         netmask  255.255.255.0
 182         bridge_ports eno1
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan2
 188 iface vxlan2 inet manual
 189         vxlan-id 2
 190         vxlan_remoteip 192.168.0.2
 191         vxlan_remoteip 192.168.0.3
 192
 193
 194 auto vmbr2
 195 iface vmbr2 inet manual
 196         bridge_ports vxlan2
 197         bridge_stp off
 198         bridge_fd 0
 199
 200
 201 auto vxlan3
 202 iface vxlan2 inet manual
 203         vxlan-id 3
 204         vxlan_remoteip 192.168.0.2
 205         vxlan_remoteip 192.168.0.3
 206
 207
 208 auto vmbr3
 209 iface vmbr3 inet manual
 210         bridge_ports vxlan3
 211         bridge_stp off
 212         bridge_fd 0
 213 ----
 214
 215
 216 * node2
 217
 218 ----
 219 auto eno1
 220 iface eno1 inet manual
 221         mtu 1550
 222
 223 auto vmbr0
 224 iface vmbr0 inet static
 225         address 192.168.0.2
 226         netmask  255.255.255.0
 227         bridge_ports eno1
 228         bridge_stp off
 229         bridge_fd 0
 230
 231 auto vxlan2
 232 iface vxlan2 inet manual
 233         vxlan-id 2
 234         vxlan_remoteip 192.168.0.1
 235         vxlan_remoteip 192.168.0.3
 236
 237
 238
 239 auto vmbr2
 240 iface vmbr2 inet manual
 241         bridge_ports vxlan2
 242         bridge_stp off
 243         bridge_fd 0
 244
 245 auto vxlan3
 246 iface vxlan2 inet manual
 247         vxlan-id 3
 248         vxlan_remoteip 192.168.0.1
 249         vxlan_remoteip 192.168.0.3
 250
 251
 252 auto vmbr3
 253 iface vmbr3 inet manual
 254         bridge_ports vxlan3
 255         bridge_stp off
 256         bridge_fd 0
 257 ----
 258
 259
 260 * node3
 261
 262 ----
 263 auto eno1
 264 iface eno1 inet manual
 265         mtu 1550
 266
 267 auto vmbr0
 268 iface vmbr0 inet static
 269         address 192.168.0.3
 270         netmask  255.255.255.0
 271         bridge_ports eno1
 272         bridge_stp off
 273         bridge_fd 0
 274
 275 auto vxlan2
 276 iface vxlan2 inet manual
 277         vxlan-id 2
 278         vxlan_remoteip 192.168.0.2
 279         vxlan_remoteip 192.168.0.3
 280
 281
 282
 283 auto vmbr2
 284 iface vmbr2 inet manual
 285         bridge_ports vxlan2
 286         bridge_stp off
 287         bridge_fd 0
 288
 289 auto vxlan3
 290 iface vxlan2 inet manual
 291         vxlan-id 3
 292         vxlan_remoteip 192.168.0.2
 293         vxlan_remoteip 192.168.0.3
 294
 295
 296 auto vmbr3
 297 iface vmbr3 inet manual
 298         bridge_ports vxlan3
 299         bridge_stp off
 300         bridge_fd 0
 301 ----
 302
 303
 304 bgp-evpn
 305 ^^^^^^^^
 306
 307 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 308 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 309
 310 The control plane used here is FRR, a bgp routing software.
 311 Each node in the proxmox cluster peer with each others nodes.
 312 For bigger networks, or multiple proxmox clusters,
 313 it's possible to use external bgp route reflector servers.
 314
 315 * node1
 316
 317 ----
 318 auto eno1
 319 iface eno1 inet manual
 320         mtu 1550
 321
 322 auto vmbr0
 323 iface vmbr0 inet static
 324         address 192.168.0.1
 325         netmask 255.255.255.0
 326         bridge_ports eno1
 327         bridge_stp off
 328         bridge_fd 0
 329
 330 auto vxlan2
 331 iface vxlan2 inet manual
 332         vxlan-id 2
 333         vxlan-local-tunnelip 192.168.0.1
 334         bridge-learning off
 335         bridge-arp-nd-suppress on
 336         bridge-unicast-flood off
 337         bridge-multicast-flood off
 338
 339
 340 auto vmbr2
 341 iface vmbr2 inet manual
 342         bridge_ports vxlan2
 343         bridge_stp off
 344         bridge_fd 0
 345
 346
 347 auto vxlan3
 348 iface vxlan3 inet manual
 349         vxlan-id 3
 350         vxlan-local-tunnelip 192.168.0.1
 351         bridge-learning off
 352         bridge-arp-nd-suppress on
 353         bridge-unicast-flood off
 354         bridge-multicast-flood off
 355
 356
 357 auto vmbr3
 358 iface vmbr3 inet manual
 359         bridge_ports vxlan3
 360         bridge_stp off
 361         bridge_fd 0
 362 ----
 363
 364
 365 /etc/frr/frr.conf
 366
 367 ----
 368 router bgp 1234
 369  no bgp default ipv4-unicast
 370  no bgp default ipv6-unicast
 371  coalesce-time 1000
 372  neighbor 192.168.0.2 remote-as 1234
 373  neighbor 192.168.0.3 remote-as 1234
 374  !
 375  address-family l2vpn evpn
 376   neighbor 192.168.0.2 activate
 377   neighbor 192.168.0.3 activate
 378   advertise-all-vni
 379  exit-address-family
 380 !
 381 line vty
 382 !
 383 ----
 384
 385
 386 * node2
 387
 388 ----
 389 auto eno1
 390 iface eno1 inet manual
 391         mtu 1550
 392
 393 auto vmbr0
 394 iface vmbr0 inet static
 395         address 192.168.0.2
 396         netmask 255.255.255.0
 397         bridge_ports eno1
 398         bridge_stp off
 399         bridge_fd 0
 400
 401 auto vxlan2
 402 iface vxlan2 inet manual
 403         vxlan-id 2
 404         vxlan-local-tunnelip 192.168.0.2
 405         bridge-learning off
 406         bridge-arp-nd-suppress on
 407         bridge-unicast-flood off
 408         bridge-multicast-flood off
 409
 410
 411 auto vmbr2
 412 iface vmbr2 inet manual
 413         bridge_ports vxlan2
 414         bridge_stp off
 415         bridge_fd 0
 416
 417 auto vxlan3
 418 iface vxlan3 inet manual
 419         vxlan-id 3
 420         vxlan-local-tunnelip 192.168.0.2
 421         bridge-learning off
 422         bridge-arp-nd-suppress on
 423         bridge-unicast-flood off
 424         bridge-multicast-flood off
 425
 426
 427 auto vmbr3
 428 iface vmbr3 inet manual
 429         bridge_ports vxlan3
 430         bridge_stp off
 431         bridge_fd 0
 432 ----
 433
 434
 435 /etc/frr/frr.conf
 436
 437 ----
 438 router bgp 1234
 439  no bgp default ipv4-unicast
 440  no bgp default ipv6-unicast
 441  coalesce-time 1000
 442  neighbor 192.168.0.1 remote-as 1234
 443  neighbor 192.168.0.3 remote-as 1234
 444  !
 445  address-family l2vpn evpn
 446   neighbor 192.168.0.1 activate
 447   neighbor 192.168.0.3 activate
 448   advertise-all-vni
 449  exit-address-family
 450 !
 451 line vty
 452 !
 453 ----
 454
 455
 456 * node3
 457
 458 ----
 459 auto eno1
 460 iface eno1 inet manual
 461         mtu 1550
 462
 463 auto vmbr0
 464 iface vmbr0 inet static
 465         address 192.168.0.2
 466         netmask 255.255.255.0
 467         bridge_ports eno1
 468         bridge_stp off
 469         bridge_fd 0
 470
 471 auto vxlan2
 472 iface vxlan2 inet manual
 473         vxlan-id 2
 474         vxlan-local-tunnelip 192.168.0.3
 475         bridge-learning off
 476         bridge-arp-nd-suppress on
 477         bridge-unicast-flood off
 478         bridge-multicast-flood off
 479
 480
 481 auto vmbr2
 482 iface vmbr2 inet manual
 483         bridge_ports vxlan2
 484         bridge_stp off
 485         bridge_fd 0
 486
 487 auto vxlan3
 488 iface vxlan3 inet manual
 489         vxlan-id 3
 490         vxlan-local-tunnelip 192.168.0.3
 491         bridge-learning off
 492         bridge-arp-nd-suppress on
 493         bridge-unicast-flood off
 494         bridge-multicast-flood off
 495
 496
 497 auto vmbr3
 498 iface vmbr3 inet manual
 499         bridge_ports vxlan3
 500         bridge_stp off
 501         bridge_fd 0
 502 ----
 503
 504
 505 /etc/frr/frr.conf
 506
 507
 508 ----
 509 router bgp 1234
 510  no bgp default ipv4-unicast
 511  no bgp default ipv6-unicast
 512  coalesce-time 1000
 513  neighbor 192.168.0.1 remote-as 1234
 514  neighbor 192.168.0.2 remote-as 1234
 515  !
 516  address-family l2vpn evpn
 517   neighbor 192.168.0.1 activate
 518   neighbor 192.168.0.2 activate
 519   advertise-all-vni
 520  exit-address-family
 521 !
 522 line vty
 523 !
 524 ----
 525
 526 VXLAN layer3 routing with anycast gateway
 527 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 528
 529 With this need, each vmbr bridge will be the gateway for the vm.
 530 Same vmbr on different node, will have same ip address and same mac address,
 531 to have working vm live migration and no network disruption.
 532
 533 VXLAN layer3 routing only work with FRR and non-aware bridge.
 534 (vlan aware bridge support is buggy currently).
 535
 536 asymmetric model
 537 ^^^^^^^^^^^^^^^^
 538
 539 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 540
 541 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 542 but only bridging on the egress.
 543 This results in bi-directional VXLAN traffic traveling on different VNIs
 544 in each direction (always the destination VNI) across the routed infrastructure.
 545
 546 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 547
 548 * node1
 549
 550 ----
 551 auto eno1
 552 iface eno1 inet manual
 553         mtu 1550
 554
 555 auto vmbr0
 556 iface vmbr0 inet static
 557          address 192.168.0.1
 558          netmask  255.255.255.0
 559          bridge_ports eno1
 560          bridge_stp off
 561          bridge_fd 0
 562
 563 auto vxlan2
 564 iface vxlan2 inet manual
 565         vxlan-id 2
 566         vxlan-local-tunnelip 192.168.0.1
 567         bridge-learning off
 568         bridge-arp-nd-suppress on
 569         bridge-unicast-flood off
 570         bridge-multicast-flood off
 571
 572
 573 auto vmbr2
 574 iface vmbr2 inet static
 575         address 10.0.2.254
 576         netmask 255.255.255.0
 577         hwaddress 44:39:39:FF:40:94
 578         bridge_ports vxlan2
 579         bridge_stp off
 580         bridge_fd 0
 581         ip-forward on
 582         ip6-forward on
 583         arp-accept on
 584
 585 auto vxlan3
 586 iface vxlan3 inet manual
 587         vxlan-id 3
 588         vxlan-local-tunnelip 192.168.0.1
 589         bridge-learning off
 590         bridge-arp-nd-suppress on
 591         bridge-unicast-flood off
 592         bridge-multicast-flood off
 593
 594
 595 auto vmbr3
 596 iface vmbr3 inet static
 597         address 10.0.3.254
 598         netmask 255.255.255.0
 599         hwaddress 44:39:39:FF:40:94
 600         bridge_ports vxlan3
 601         bridge_stp off
 602         bridge_fd 0
 603         ip-forward on
 604         ip6-forward on
 605         arp-accept on
 606 ----
 607
 608
 609 frr.conf
 610
 611 ----
 612 router bgp 1234
 613  bgp router-id 192.168.0.1
 614  no bgp default ipv4-unicast
 615  no bgp default ipv6-unicast
 616  coalesce-time 1000
 617  neighbor 192.168.0.2 remote-as 1234
 618  neighbor 192.168.0.3 remote-as 1234
 619  !
 620  address-family l2vpn evpn
 621   neighbor 192.168.0.2 activate
 622   neighbor 192.168.0.3 activate
 623   advertise-all-vni
 624  exit-address-family
 625 !
 626 line vty
 627 !
 628 ----
 629
 630
 631 * node2
 632
 633 ----
 634 auto eno1
 635 iface eno1 inet manual
 636         mtu 1550
 637
 638 auto vmbr0
 639 iface vmbr0 inet static
 640          address 192.168.0.2
 641          netmask  255.255.255.0
 642          bridge_ports eno1
 643          bridge_stp off
 644          bridge_fd 0
 645
 646 auto vxlan2
 647 iface vxlan2 inet manual
 648         vxlan-id 2
 649         vxlan-local-tunnelip 192.168.0.2
 650         bridge-learning off
 651         bridge-arp-nd-suppress on
 652         bridge-unicast-flood off
 653         bridge-multicast-flood off
 654
 655
 656 auto vmbr2
 657 iface vmbr2 inet static
 658         address 10.0.2.254
 659         netmask 255.255.255.0
 660         hwaddress 44:39:39:FF:40:94
 661         bridge_ports vxlan2
 662         bridge_stp off
 663         bridge_fd 0
 664         ip-forward on
 665         ip6-forward on
 666         arp-accept on
 667
 668
 669 auto vxlan3
 670 iface vxlan3 inet manual
 671         vxlan-id 3
 672         vxlan-local-tunnelip 192.168.0.2
 673         bridge-learning off
 674         bridge-arp-nd-suppress on
 675         bridge-unicast-flood off
 676         bridge-multicast-flood off
 677
 678
 679 auto vmbr3
 680 iface vmbr3 inet static
 681         address 10.0.3.254
 682         netmask 255.255.255.0
 683         hwaddress 44:39:39:FF:40:94
 684         bridge_ports vxlan3
 685         bridge_stp off
 686         bridge_fd 0
 687         ip-forward on
 688         ip6-forward on
 689         arp-accept on
 690 ----
 691
 692
 693 frr.conf
 694
 695 ----
 696 router bgp 1234
 697  bgp router-id 192.168.0.2
 698  no bgp default ipv4-unicast
 699  no bgp default ipv6-unicast
 700  coalesce-time 1000
 701  neighbor 192.168.0.1 remote-as 1234
 702  neighbor 192.168.0.3 remote-as 1234
 703  !
 704  address-family l2vpn evpn
 705   neighbor 192.168.0.1 activate
 706   neighbor 192.168.0.3 activate
 707   advertise-all-vni
 708  exit-address-family
 709 !
 710 line vty
 711 !
 712 ----
 713
 714
 715 * node3
 716
 717 ----
 718 auto eno1
 719 iface eno1 inet manual
 720         mtu 1550
 721
 722 auto vmbr0
 723 iface vmbr0 inet static
 724          address 192.168.0.3
 725          netmask  255.255.255.0
 726          bridge_ports eno1
 727          bridge_stp off
 728          bridge_fd 0
 729
 730 auto vxlan2
 731 iface vxlan2 inet manual
 732         vxlan-id 2
 733         vxlan-local-tunnelip 192.168.0.3
 734         bridge-learning off
 735         bridge-arp-nd-suppress on
 736         bridge-unicast-flood off
 737         bridge-multicast-flood off
 738
 739
 740 auto vmbr2
 741 iface vmbr2 inet static
 742         address 10.0.2.254
 743         netmask 255.255.255.0
 744         hwaddress 44:39:39:FF:40:94
 745         bridge_ports vxlan2
 746         bridge_stp off
 747         bridge_fd 0
 748         ip-forward on
 749         ip6-forward on
 750         arp-accept on
 751
 752 auto vxlan3
 753 iface vxlan3 inet manual
 754         vxlan-id 3
 755         vxlan-local-tunnelip 192.168.0.3
 756         bridge-learning off
 757         bridge-arp-nd-suppress on
 758         bridge-unicast-flood off
 759         bridge-multicast-flood off
 760
 761 auto vmbr3
 762 iface vmbr3 inet static
 763         address 10.0.3.254
 764         netmask 255.255.255.0
 765         hwaddress 44:39:39:FF:40:94
 766         bridge_ports vxlan3
 767         bridge_stp off
 768         bridge_fd 0
 769         ip-forward on
 770         ip6-forward on
 771         arp-accept on
 772 ----
 773
 774
 775 frr.conf
 776
 777 ----
 778 router bgp 1234
 779  bgp router-id 192.168.0.3
 780  no bgp default ipv4-unicast
 781  no bgp default ipv6-unicast
 782  coalesce-time 1000
 783  neighbor 192.168.0.1 remote-as 1234
 784  neighbor 192.168.0.2 remote-as 1234
 785  !
 786  address-family l2vpn evpn
 787   neighbor 192.168.0.1 activate
 788   neighbor 192.168.0.2 activate
 789   advertise-all-vni
 790  exit-address-family
 791 !
 792 line vty
 793 !
 794 ----
 795
 796
 797 symmetric model
 798 ^^^^^^^^^^^^^^^
 799
 800 With this model, you don't need to have all vxlan on all nodes.
 801 This model will also be needed to route traffic to an external router.
 802
 803 The symmetric model routes and bridges on both the ingress and the egress leafs.
 804 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 805 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 806 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 807 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 808
 809 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 810
 811 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 812
 813 * node1
 814
 815 ----
 816 auto vrf1
 817 iface vrf1
 818     vrf-table auto
 819
 820 auto eno1
 821 iface eno1 inet manual
 822         mtu 1550
 823
 824 auto vmbr0
 825 iface vmbr0 inet static
 826          address 192.168.0.1
 827          netmask  255.255.255.0
 828          bridge_ports eno1
 829          bridge_stp off
 830          bridge_fd 0
 831
 832 auto vxlan2
 833 iface vxlan2 inet manual
 834         vxlan-id 2
 835         vxlan-local-tunnelip 192.168.0.1
 836         bridge-learning off
 837         bridge-arp-nd-suppress on
 838         bridge-unicast-flood off
 839         bridge-multicast-flood off
 840
 841 auto vmbr2
 842 iface vmbr2 inet static
 843         bridge_ports vxlan2
 844         bridge_stp off
 845         bridge_fd 0
 846         address 10.0.2.254
 847         netmask 255.255.255.0
 848         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 849         vrf vrf1
 850         ip-forward on
 851         ip6-forward on
 852         arp-accept on
 853
 854 auto vxlan3
 855 iface vxlan3 inet manual
 856         vxlan-id 3
 857         vxlan-local-tunnelip 192.168.0.1
 858         bridge-learning off
 859         bridge-arp-nd-suppress on
 860         bridge-unicast-flood off
 861         bridge-multicast-flood off
 862
 863 auto vmbr3
 864 iface vmbr3 inet static
 865         bridge_ports vxlan3
 866         bridge_stp off
 867         bridge_fd 0
 868         address 10.0.3.254
 869         netmask 255.255.255.0
 870         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 871         vrf vrf1
 872         ip-forward on
 873         ip6-forward on
 874         arp-accept on
 875
 876 #interconnect vxlan-vfr l3vni
 877 auto vxlan4000
 878 iface vxlan4000 inet manual
 879         vxlan-id 4000
 880         vxlan-local-tunnelip 192.168.0.1
 881         bridge-learning off
 882         bridge-arp-nd-suppress on
 883         bridge-unicast-flood off
 884         bridge-multicast-flood off
 885
 886
 887 auto vmbr4000
 888 iface vmbr4000 inet manual
 889         bridge_ports vxlan4000
 890         bridge_stp off
 891         bridge_fd 0
 892         vrf vrf1
 893 ----
 894
 895 frr.conf
 896
 897 ----
 898 vrf vrf1
 899  vni 4000
 900  exit-vrf
 901 !
 902 router bgp 1234
 903  bgp router-id 192.168.0.1
 904  no bgp default ipv4-unicast
 905  no bgp default ipv6-unicast
 906  coalesce-time 1000
 907  neighbor 192.168.0.2 remote-as 1234
 908  neighbor 192.168.0.3 remote-as 1234
 909  !
 910  address-family l2vpn evpn
 911   neighbor 192.168.0.2 activate
 912   neighbor 192.168.0.3 activate
 913   advertise-all-vni
 914  exit-address-family
 915 !
 916 line vty
 917 !
 918 ----
 919
 920
 921 * node2
 922
 923 ----
 924 auto vrf1
 925 iface vrf1
 926     vrf-table auto
 927
 928 auto eno1
 929 iface eno1 inet manual
 930         mtu 1550
 931
 932 auto vmbr0
 933 iface vmbr0 inet static
 934          address 192.168.0.2
 935          netmask  255.255.255.0
 936          bridge_ports eno1
 937          bridge_stp off
 938          bridge_fd 0
 939
 940 auto vxlan2
 941 iface vxlan2 inet manual
 942         vxlan-id 2
 943         vxlan-local-tunnelip 192.168.0.2
 944         bridge-learning off
 945         bridge-arp-nd-suppress on
 946         bridge-unicast-flood off
 947         bridge-multicast-flood off
 948
 949 auto vmbr2
 950 iface vmbr2 inet static
 951         bridge_ports vxlan2
 952         bridge_stp off
 953         bridge_fd 0
 954         address 10.0.2.254
 955         netmask 255.255.255.0
 956         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 957         vrf vrf1
 958         ip-forward on
 959         ip6-forward on
 960         arp-accept on
 961
 962 auto vxlan3
 963 iface vxlan3 inet manual
 964         vxlan-id 3
 965         vxlan-local-tunnelip 192.168.0.2
 966         bridge-learning off
 967         bridge-arp-nd-suppress on
 968         bridge-unicast-flood off
 969         bridge-multicast-flood off
 970
 971 auto vmbr3
 972 iface vmbr3 inet static
 973         bridge_ports vxlan3
 974         bridge_stp off
 975         bridge_fd 0
 976         address 10.0.3.254
 977         netmask 255.255.255.0
 978         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 979         vrf vrf1
 980         ip-forward on
 981         ip6-forward on
 982         arp-accept on
 983
 984 #interconnect vxlan-vfr l3vni
 985 auto vxlan4000
 986 iface vxlan4000 inet manual
 987         vxlan-id 4000
 988         vxlan-local-tunnelip 192.168.0.2
 989         bridge-learning off
 990         bridge-arp-nd-suppress on
 991         bridge-unicast-flood off
 992         bridge-multicast-flood off
 993
 994
 995 auto vmbr4000
 996 iface vmbr4000 inet manual
 997         bridge_ports vxlan4000
 998         bridge_stp off
 999         bridge_fd 0
1000         vrf vrf1
1001 ----
1002
1003
1004 frr.conf
1005
1006 ----
1007 vrf vrf1
1008  vni 4000
1009  exit-vrf
1010 !
1011 router bgp 1234
1012  bgp router-id 192.168.0.2
1013  no bgp default ipv4-unicast
1014  no bgp default ipv6-unicast
1015  coalesce-time 1000
1016  neighbor 192.168.0.1 remote-as 1234
1017  neighbor 192.168.0.3 remote-as 1234
1018  !
1019  address-family l2vpn evpn
1020   neighbor 192.168.0.1 activate
1021   neighbor 192.168.0.3 activate
1022   advertise-all-vni
1023  exit-address-family
1024 !
1025 line vty
1026 !
1027 ----
1028
1029
1030 * node3
1031
1032 ----
1033 auto vrf1
1034 iface vrf1
1035     vrf-table auto
1036
1037 auto eno1
1038 iface eno1 inet manual
1039         mtu 1550
1040
1041 auto vmbr0
1042 iface vmbr0 inet static
1043          address 192.168.0.3
1044          netmask  255.255.255.0
1045          bridge_ports eno1
1046          bridge_stp off
1047          bridge_fd 0
1048
1049 auto vxlan2
1050 iface vxlan2 inet manual
1051         vxlan-id 2
1052         vxlan-local-tunnelip 192.168.0.3
1053         bridge-learning off
1054         bridge-arp-nd-suppress on
1055         bridge-unicast-flood off
1056         bridge-multicast-flood off
1057
1058 auto vmbr2
1059 iface vmbr2 inet static
1060         bridge_ports vxlan2
1061         bridge_stp off
1062         bridge_fd 0
1063         address 10.0.2.254
1064         netmask 255.255.255.0
1065         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1066         vrf vrf1
1067         ip-forward on
1068         ip6-forward on
1069         arp-accept on
1070
1071 auto vxlan3
1072 iface vxlan3 inet manual
1073         vxlan-id 3
1074         vxlan-local-tunnelip 192.168.0.3
1075         bridge-learning off
1076         bridge-arp-nd-suppress on
1077         bridge-unicast-flood off
1078         bridge-multicast-flood off
1079
1080 auto vmbr3
1081 iface vmbr3 inet static
1082         bridge_ports vxlan3
1083         bridge_stp off
1084         bridge_fd 0
1085         address 10.0.3.254
1086         netmask 255.255.255.0
1087         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1088         vrf vrf1
1089         ip-forward on
1090         ip6-forward on
1091         arp-accept on
1092
1093 #interconnect vxlan-vfr l3vni
1094 auto vxlan4000
1095 iface vxlan4000 inet manual
1096         vxlan-id 4000
1097         vxlan-local-tunnelip 192.168.0.3
1098         bridge-learning off
1099         bridge-arp-nd-suppress on
1100         bridge-unicast-flood off
1101         bridge-multicast-flood off
1102
1103
1104 auto vmbr4000
1105 iface vmbr4000 inet manual
1106         bridge_ports vxlan4000
1107         bridge_stp off
1108         bridge_fd 0
1109         vrf vrf1
1110 ----
1111
1112
1113 frr.conf
1114
1115 ----
1116 vrf vrf1
1117  vni 4000
1118  exit-vrf
1119 !
1120 router bgp 1234
1121  bgp router-id 192.168.0.3
1122  no bgp default ipv4-unicast
1123  no bgp default ipv6-unicast
1124  coalesce-time 1000
1125  neighbor 192.168.0.1 remote-as 1234
1126  neighbor 192.168.0.2 remote-as 1234
1127  !
1128  address-family l2vpn evpn
1129   neighbor 192.168.0.1 activate
1130   neighbor 192.168.0.2 activate
1131   advertise-all-vni
1132  exit-address-family
1133 !
1134 line vty
1135 !
1136 ----
1137
1138 VXLAN layer3 routing with anycast gateway + routing to outside with external router with static default gw
1139 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1140 Routing to outside need the symmetric model.
1141
1142 1 gateway node
1143 ^^^^^^^^^^^^^^
1144 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1145 This node announce the default gw in vrf1 (default originate) and forward to his own default gateway (192.168.0.254) (no bgp between router and node1)
1146
1147
1148 *node1
1149
1150 ----
1151 auto vrf1
1152 iface vrf1
1153     vrf-table auto
1154
1155 auto eno1
1156 iface eno1 inet manual
1157         mtu 1550
1158
1159 auto vmbr0
1160 iface vmbr0 inet static
1161          address 192.168.0.1
1162          netmask  255.255.255.0
1163          gateway 192.168.0.254
1164          bridge_ports eno1
1165          bridge_stp off
1166          bridge_fd 0
1167          ip-forward on
1168          ip6-forward on
1169
1170 auto vxlan2
1171 iface vxlan2 inet manual
1172         vxlan-id 2
1173         vxlan-local-tunnelip 192.168.0.1
1174         bridge-learning off
1175         bridge-arp-nd-suppress on
1176         bridge-unicast-flood off
1177         bridge-multicast-flood off
1178
1179 auto vmbr2
1180 iface vmbr2 inet static
1181         bridge_ports vxlan2
1182         bridge_stp off
1183         bridge_fd 0
1184         address 10.0.2.254
1185         netmask 255.255.255.0
1186         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1187         vrf vrf1
1188         ip-forward on
1189         ip6-forward on
1190         arp-accept on
1191
1192 auto vxlan3
1193 iface vxlan3 inet manual
1194         vxlan-id 3
1195         vxlan-local-tunnelip 192.168.0.1
1196         bridge-learning off
1197         bridge-arp-nd-suppress on
1198         bridge-unicast-flood off
1199         bridge-multicast-flood off
1200
1201 auto vmbr3
1202 iface vmbr3 inet static
1203         bridge_ports vxlan3
1204         bridge_stp off
1205         bridge_fd 0
1206         address 10.0.3.254
1207         netmask 255.255.255.0
1208         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1209         vrf vrf1
1210         ip-forward on
1211         ip6-forward on
1212         arp-accept on
1213
1214 #interconnect vxlan-vfr l3vni
1215 auto vxlan4000
1216 iface vxlan4000 inet manual
1217         vxlan-id 4000
1218         vxlan-local-tunnelip 192.168.0.1
1219         bridge-learning off
1220         bridge-arp-nd-suppress on
1221         bridge-unicast-flood off
1222         bridge-multicast-flood off
1223
1224 auto vmbr4000
1225 iface vmbr4000 inet manual
1226         bridge_ports vxlan4000
1227         bridge_stp off
1228         bridge_fd 0
1229         vrf vrf1
1230 ----
1231
1232
1233 frr.conf
1234
1235 ----
1236 vrf vrf1
1237  vni 4000
1238  exit-vrf
1239 !
1240 router bgp 1234
1241  bgp router-id 192.168.0.1
1242  no bgp default ipv4-unicast
1243  no bgp default ipv6-unicast
1244  coalesce-time 1000
1245  neighbor 192.168.0.2 remote-as 1234
1246  neighbor 192.168.0.3 remote-as 1234
1247  !
1248  address-family ipv4 unicast
1249   import vrf vrf1
1250  exit-address-family
1251  !
1252  address-family ipv6 unicast
1253   import vrf vrf1
1254  exit-address-family
1255  !
1256  address-family l2vpn evpn
1257   neighbor 192.168.0.2 activate
1258   neighbor 192.168.0.3 activate
1259   advertise-all-vni
1260  exit-address-family
1261 !
1262 router bgp 1234 vrf vrf1
1263 !
1264  address-family ipv4 unicast
1265   redistribute connected
1266  exit-address-family
1267  !
1268  address-family ipv6 unicast
1269   redistribute connected
1270  exit-address-family
1271  !
1272  address-family l2vpn evpn
1273   default-originate ipv4
1274   default-originate ipv6
1275  exit-address-family
1276 !
1277 line vty
1278 !
1279 ----
1280
1281
1282 * node2
1283
1284 ----
1285 auto vrf1
1286 iface vrf1
1287     vrf-table auto
1288
1289 auto eno1
1290 iface eno1 inet manual
1291         mtu 1550
1292
1293 auto vmbr0
1294 iface vmbr0 inet static
1295          address 192.168.0.2
1296          netmask  255.255.255.0
1297          bridge_ports eno1
1298          bridge_stp off
1299          bridge_fd 0
1300
1301 auto vxlan2
1302 iface vxlan2 inet manual
1303         vxlan-id 2
1304         vxlan-local-tunnelip 192.168.0.2
1305         bridge-learning off
1306         bridge-arp-nd-suppress on
1307         bridge-unicast-flood off
1308         bridge-multicast-flood off
1309
1310 auto vmbr2
1311 iface vmbr2 inet static
1312         bridge_ports vxlan2
1313         bridge_stp off
1314         bridge_fd 0
1315         address 10.0.2.254
1316         netmask 255.255.255.0
1317         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1318         vrf vrf1
1319         ip-forward on
1320         ip6-forward on
1321         arp-accept on
1322
1323 auto vxlan3
1324 iface vxlan3 inet manual
1325         vxlan-id 3
1326         vxlan-local-tunnelip 192.168.0.2
1327         bridge-learning off
1328         bridge-arp-nd-suppress on
1329         bridge-unicast-flood off
1330         bridge-multicast-flood off
1331
1332 auto vmbr3
1333 iface vmbr3 inet static
1334         bridge_ports vxlan3
1335         bridge_stp off
1336         bridge_fd 0
1337         address 10.0.3.254
1338         netmask 255.255.255.0
1339         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1340         vrf vrf1
1341         ip-forward on
1342         ip6-forward on
1343         arp-accept on
1344
1345 #interconnect vxlan-vfr l3vni
1346 auto vxlan4000
1347 iface vxlan4000 inet manual
1348         vxlan-id 4000
1349         vxlan-local-tunnelip 192.168.0.2
1350         bridge-learning off
1351         bridge-arp-nd-suppress on
1352         bridge-unicast-flood off
1353         bridge-multicast-flood off
1354
1355
1356 auto vmbr4000
1357 iface vmbr4000 inet manual
1358         bridge_ports vxlan4000
1359         bridge_stp off
1360         bridge_fd 0
1361         vrf vrf1
1362 ----
1363
1364
1365 frr.conf
1366
1367 ----
1368 vrf vrf1
1369  vni 4000
1370  exit-vrf
1371 !
1372 router bgp 1234
1373  bgp router-id 192.168.0.2
1374  no bgp default ipv4-unicast
1375  no bgp default ipv6-unicast
1376  coalesce-time 1000
1377  neighbor 192.168.0.1 remote-as 1234
1378  neighbor 192.168.0.3 remote-as 1234
1379  !
1380  address-family l2vpn evpn
1381   neighbor 192.168.0.1 activate
1382   neighbor 192.168.0.3 activate
1383   advertise-all-vni
1384  exit-address-family
1385 !
1386 line vty
1387 !
1388 ----
1389
1390
1391 * node3
1392
1393 ----
1394 auto vrf1
1395 iface vrf1
1396     vrf-table auto
1397
1398 auto eno1
1399 iface eno1 inet manual
1400         mtu 1550
1401
1402 auto vmbr0
1403 iface vmbr0 inet static
1404          address 192.168.0.3
1405          netmask  255.255.255.0
1406          bridge_ports eno1
1407          bridge_stp off
1408          bridge_fd 0
1409
1410 auto vxlan2
1411 iface vxlan2 inet manual
1412         vxlan-id 2
1413         vxlan-local-tunnelip 192.168.0.3
1414         bridge-learning off
1415         bridge-arp-nd-suppress on
1416         bridge-unicast-flood off
1417         bridge-multicast-flood off
1418
1419 auto vmbr2
1420 iface vmbr2 inet static
1421         bridge_ports vxlan2
1422         bridge_stp off
1423         bridge_fd 0
1424         address 10.0.2.254
1425         netmask 255.255.255.0
1426         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1427         vrf vrf1
1428         ip-forward on
1429         ip6-forward on
1430         arp-accept on
1431
1432 auto vxlan3
1433 iface vxlan3 inet manual
1434         vxlan-id 3
1435         vxlan-local-tunnelip 192.168.0.3
1436         bridge-learning off
1437         bridge-arp-nd-suppress on
1438         bridge-unicast-flood off
1439         bridge-multicast-flood off
1440
1441 auto vmbr3
1442 iface vmbr3 inet static
1443         bridge_ports vxlan3
1444         bridge_stp off
1445         bridge_fd 0
1446         address 10.0.3.254
1447         netmask 255.255.255.0
1448         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1449         vrf vrf1
1450         ip-forward on
1451         ip6-forward on
1452         arp-accept on
1453
1454 #interconnect vxlan-vfr l3vni
1455 auto vxlan4000
1456 iface vxlan4000 inet manual
1457         vxlan-id 4000
1458         vxlan-local-tunnelip 192.168.0.3
1459         bridge-learning off
1460         bridge-arp-nd-suppress on
1461         bridge-unicast-flood off
1462         bridge-multicast-flood off
1463
1464
1465 auto vmbr4000
1466 iface vmbr4000 inet manual
1467         bridge_ports vxlan4000
1468         bridge_stp off
1469         bridge_fd 0
1470         vrf vrf1
1471 ----
1472
1473
1474 frr.conf
1475
1476 ----
1477 vrf vrf1
1478  vni 4000
1479  exit-vrf
1480 !
1481 router bgp 1234
1482  bgp router-id 192.168.0.3
1483  no bgp default ipv4-unicast
1484  no bgp default ipv6-unicast
1485  coalesce-time 1000
1486  neighbor 192.168.0.1 remote-as 1234
1487  neighbor 192.168.0.2 remote-as 1234
1488  !
1489  address-family l2vpn evpn
1490   neighbor 192.168.0.1 activate
1491   neighbor 192.168.0.2 activate
1492   advertise-all-vni
1493  exit-address-family
1494 !
1495 line vty
1496 !
1497 ----
1498
1499 multiple gateway nodes
1500 ^^^^^^^^^^^^^^^^^^^^^^
1501 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1502 All nodes have a a default gw to the external router (192.168.0.254) (no bgp between router and node1)
1503 and announce this default gw in the vrf (default originate)
1504 The external router have ecmp routes to all proxmox nodes.(balancing).
1505 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1506 vxlan the packet to final destination.
1507
1508 If you have multiple gateway nodes, disable rp_filter as packet could incoming in a 1 node, and outgoing
1509 to another node.
1510
1511 sysctl.conf tuning
1512 -----
1513 net.ipv4.conf.default.rp_filter=0
1514 net.ipv4.conf.all.rp_filter=0
1515 -----
1516
1517
1518 *node1
1519
1520 ----
1521 auto vrf1
1522 iface vrf1
1523     vrf-table auto
1524
1525 auto eno1
1526 iface eno1 inet manual
1527         mtu 1550
1528
1529 auto vmbr0
1530 iface vmbr0 inet static
1531          address 192.168.0.1
1532          netmask  255.255.255.0
1533          gateway 192.168.0.254
1534          bridge_ports eno1
1535          bridge_stp off
1536          bridge_fd 0
1537          ip-forward on
1538          ip6-forward on
1539
1540 auto vxlan2
1541 iface vxlan2 inet manual
1542         vxlan-id 2
1543         vxlan-local-tunnelip 192.168.0.1
1544         bridge-learning off
1545         bridge-arp-nd-suppress on
1546         bridge-unicast-flood off
1547         bridge-multicast-flood off
1548
1549 auto vmbr2
1550 iface vmbr2 inet static
1551         bridge_ports vxlan2
1552         bridge_stp off
1553         bridge_fd 0
1554         address 10.0.2.254
1555         netmask 255.255.255.0
1556         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1557         vrf vrf1
1558         ip-forward on
1559         ip6-forward on
1560         arp-accept on
1561
1562 auto vxlan3
1563 iface vxlan3 inet manual
1564         vxlan-id 3
1565         vxlan-local-tunnelip 192.168.0.1
1566         bridge-learning off
1567         bridge-arp-nd-suppress on
1568         bridge-unicast-flood off
1569         bridge-multicast-flood off
1570
1571 auto vmbr3
1572 iface vmbr3 inet static
1573         bridge_ports vxlan3
1574         bridge_stp off
1575         bridge_fd 0
1576         address 10.0.3.254
1577         netmask 255.255.255.0
1578         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1579         vrf vrf1
1580         ip-forward on
1581         ip6-forward on
1582         arp-accept on
1583
1584 #interconnect vxlan-vfr l3vni
1585 auto vxlan4000
1586 iface vxlan4000 inet manual
1587         vxlan-id 4000
1588         vxlan-local-tunnelip 192.168.0.1
1589         bridge-learning off
1590         bridge-arp-nd-suppress on
1591         bridge-unicast-flood off
1592         bridge-multicast-flood off
1593
1594 auto vmbr4000
1595 iface vmbr4000 inet manual
1596         bridge_ports vxlan4000
1597         bridge_stp off
1598         bridge_fd 0
1599         vrf vrf1
1600 ----
1601
1602
1603 frr.conf
1604
1605 ----
1606 vrf vrf1
1607  vni 4000
1608  exit-vrf
1609 !
1610 router bgp 1234
1611  bgp router-id 192.168.0.1
1612  no bgp default ipv4-unicast
1613  no bgp default ipv6-unicast
1614  coalesce-time 1000
1615  neighbor 192.168.0.2 remote-as 1234
1616  neighbor 192.168.0.3 remote-as 1234
1617  !
1618  address-family ipv4 unicast
1619   import vrf vrf1
1620  exit-address-family
1621  !
1622  address-family ipv6 unicast
1623   import vrf vrf1
1624  exit-address-family
1625  !
1626  address-family l2vpn evpn
1627   neighbor 192.168.0.2 activate
1628   neighbor 192.168.0.3 activate
1629   advertise-all-vni
1630  exit-address-family
1631 !
1632 router bgp 1234 vrf vrf1
1633 !
1634  address-family ipv4 unicast
1635   redistribute connected
1636  exit-address-family
1637  !
1638  address-family ipv6 unicast
1639   redistribute connected
1640  exit-address-family
1641  !
1642  address-family l2vpn evpn
1643   default-originate ipv4
1644   default-originate ipv6
1645  exit-address-family
1646 !
1647 line vty
1648 !
1649 ----
1650
1651
1652 * node2
1653
1654 ----
1655 auto vrf1
1656 iface vrf1
1657     vrf-table auto
1658
1659 auto eno1
1660 iface eno1 inet manual
1661         mtu 1550
1662
1663 auto vmbr0
1664 iface vmbr0 inet static
1665          address 192.168.0.2
1666          netmask  255.255.255.0
1667          gateway 192.168.0.254
1668          bridge_ports eno1
1669          bridge_stp off
1670          bridge_fd 0
1671          ip-forward on
1672          ip6-forward on
1673
1674 auto vxlan2
1675 iface vxlan2 inet manual
1676         vxlan-id 2
1677         vxlan-local-tunnelip 192.168.0.2
1678         bridge-learning off
1679         bridge-arp-nd-suppress on
1680         bridge-unicast-flood off
1681         bridge-multicast-flood off
1682
1683 auto vmbr2
1684 iface vmbr2 inet static
1685         bridge_ports vxlan2
1686         bridge_stp off
1687         bridge_fd 0
1688         address 10.0.2.254
1689         netmask 255.255.255.0
1690         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1691         vrf vrf1
1692         ip-forward on
1693         ip6-forward on
1694         arp-accept on
1695
1696 auto vxlan3
1697 iface vxlan3 inet manual
1698         vxlan-id 3
1699         vxlan-local-tunnelip 192.168.0.2
1700         bridge-learning off
1701         bridge-arp-nd-suppress on
1702         bridge-unicast-flood off
1703         bridge-multicast-flood off
1704
1705 auto vmbr3
1706 iface vmbr3 inet static
1707         bridge_ports vxlan3
1708         bridge_stp off
1709         bridge_fd 0
1710         address 10.0.3.254
1711         netmask 255.255.255.0
1712         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1713         vrf vrf1
1714         ip-forward on
1715         ip6-forward on
1716         arp-accept on
1717
1718 #interconnect vxlan-vfr l3vni
1719 auto vxlan4000
1720 iface vxlan4000 inet manual
1721         vxlan-id 4000
1722         vxlan-local-tunnelip 192.168.0.2
1723         bridge-learning off
1724         bridge-arp-nd-suppress on
1725         bridge-unicast-flood off
1726         bridge-multicast-flood off
1727
1728
1729 auto vmbr4000
1730 iface vmbr4000 inet manual
1731         bridge_ports vxlan4000
1732         bridge_stp off
1733         bridge_fd 0
1734         vrf vrf1
1735 ----
1736
1737
1738 frr.conf
1739
1740 ----
1741 vrf vrf1
1742  vni 4000
1743  exit-vrf
1744 !
1745 router bgp 1234
1746  bgp router-id 192.168.0.2
1747  no bgp default ipv4-unicast
1748  no bgp default ipv6-unicast
1749  coalesce-time 1000
1750  neighbor 192.168.0.1 remote-as 1234
1751  neighbor 192.168.0.3 remote-as 1234
1752  !
1753  address-family ipv4 unicast
1754   import vrf vrf1
1755  exit-address-family
1756  !
1757  address-family ipv6 unicast
1758   import vrf vrf1
1759  exit-address-family
1760  !
1761  address-family l2vpn evpn
1762   neighbor 192.168.0.1 activate
1763   neighbor 192.168.0.3 activate
1764   advertise-all-vni
1765  exit-address-family
1766 !
1767  address-family ipv4 unicast
1768   redistribute connected
1769  exit-address-family
1770  !
1771  address-family ipv6 unicast
1772   redistribute connected
1773  exit-address-family
1774  !
1775  address-family l2vpn evpn
1776   default-originate ipv4
1777   default-originate ipv6
1778  exit-address-family
1779 !
1780 line vty
1781 !
1782 ----
1783
1784
1785 * node3
1786
1787 ----
1788 auto vrf1
1789 iface vrf1
1790     vrf-table auto
1791
1792 auto eno1
1793 iface eno1 inet manual
1794         mtu 1550
1795
1796 auto vmbr0
1797 iface vmbr0 inet static
1798          address 192.168.0.3
1799          netmask  255.255.255.0
1800          gateway 192.168.0.254
1801          bridge_ports eno1
1802          bridge_stp off
1803          bridge_fd 0
1804          ip-forward on
1805          ip6-forward on
1806
1807 auto vxlan2
1808 iface vxlan2 inet manual
1809         vxlan-id 2
1810         vxlan-local-tunnelip 192.168.0.3
1811         bridge-learning off
1812         bridge-arp-nd-suppress on
1813         bridge-unicast-flood off
1814         bridge-multicast-flood off
1815
1816 auto vmbr2
1817 iface vmbr2 inet static
1818         bridge_ports vxlan2
1819         bridge_stp off
1820         bridge_fd 0
1821         address 10.0.2.254
1822         netmask 255.255.255.0
1823         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1824         vrf vrf1
1825         ip-forward on
1826         ip6-forward on
1827         arp-accept on
1828
1829 auto vxlan3
1830 iface vxlan3 inet manual
1831         vxlan-id 3
1832         vxlan-local-tunnelip 192.168.0.3
1833         bridge-learning off
1834         bridge-arp-nd-suppress on
1835         bridge-unicast-flood off
1836         bridge-multicast-flood off
1837
1838 auto vmbr3
1839 iface vmbr3 inet static
1840         bridge_ports vxlan3
1841         bridge_stp off
1842         bridge_fd 0
1843         address 10.0.3.254
1844         netmask 255.255.255.0
1845         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1846         vrf vrf1
1847         ip-forward on
1848         ip6-forward on
1849         arp-accept on
1850
1851 #interconnect vxlan-vfr l3vni
1852 auto vxlan4000
1853 iface vxlan4000 inet manual
1854         vxlan-id 4000
1855         vxlan-local-tunnelip 192.168.0.3
1856         bridge-learning off
1857         bridge-arp-nd-suppress on
1858         bridge-unicast-flood off
1859         bridge-multicast-flood off
1860
1861
1862 auto vmbr4000
1863 iface vmbr4000 inet manual
1864         bridge_ports vxlan4000
1865         bridge_stp off
1866         bridge_fd 0
1867         vrf vrf1
1868 ----
1869
1870
1871 frr.conf
1872
1873 ----
1874 vrf vrf1
1875  vni 4000
1876  exit-vrf
1877 !
1878 router bgp 1234
1879  bgp router-id 192.168.0.3
1880  no bgp default ipv4-unicast
1881  no bgp default ipv6-unicast
1882  coalesce-time 1000
1883  neighbor 192.168.0.1 remote-as 1234
1884  neighbor 192.168.0.2 remote-as 1234
1885  !
1886  address-family ipv4 unicast
1887   import vrf vrf1
1888  exit-address-family
1889  !
1890  address-family ipv6 unicast
1891   import vrf vrf1
1892  exit-address-family
1893  !
1894  address-family l2vpn evpn
1895   neighbor 192.168.0.1 activate
1896   neighbor 192.168.0.2 activate
1897   advertise-all-vni
1898  exit-address-family
1899 !
1900 router bgp 1234 vrf vrf1
1901 !
1902  address-family ipv4 unicast
1903   redistribute connected
1904  exit-address-family
1905  !
1906  address-family ipv6 unicast
1907   redistribute connected
1908  exit-address-family
1909  !
1910  address-family l2vpn evpn
1911   default-originate ipv4
1912   default-originate ipv6
1913  exit-address-family
1914 !
1915 line vty
1916 !
1917 ----
1918
1919 Note
1920 ^^^^
1921
1922 If your external router doesn't support 'ECMP static routes' to reach multiple
1923 {pve} nodes, you can setup an HA floating vip on proxmox nodes by using the
1924 Virtual Router Redundancy Protocol (VRRP).
1925
1926 In this example, we will setup an floating 192.168.0.10 IP on node1 and node2.
1927 Node1 is the primary with failover to node2 in case of outage.
1928
1929 This setup currently needs 'vrrpd' package (`apt install vrrpd`).
1930 #TODO : It should be possible to do it with frr directly with last version.
1931
1932 * node1
1933
1934 ----
1935 auto vmbr0
1936 iface vmbr0 inet static
1937          address 192.168.0.1
1938          netmask  255.255.255.0
1939          gateway 192.168.0.254
1940          bridge_ports eno1
1941          bridge_stp off
1942          bridge_fd 0
1943          vrrp-id 1
1944          vrrp-priority 1
1945          vrrp-virtual-ip 192.168.0.10
1946 ----
1947
1948 * node2
1949
1950 ----
1951 auto vmbr0
1952 iface vmbr0 inet static
1953          address 192.168.0.2
1954          netmask  255.255.255.0
1955          gateway 192.168.0.254
1956          bridge_ports eno1
1957          bridge_stp off
1958          bridge_fd 0
1959          vrrp-id 1
1960          vrrp-priority 2
1961          vrrp-virtual-ip 192.168.0.10
1962 ----
1963
1964
1965
1966 gateway node(s) with a upstream bgp router
1967 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1968 Setup is almost the same than with a static gateway, but we'll connect to an upstream bgp router.
1969
1970 example with node1 as gateway (192.168.0.1) for evpn-bgp, and an upstream bgp router (running frr too) 192.168.0.254.
1971
1972 * node1
1973
1974 frr.conf
1975 ----
1976 vrf vrf1
1977  vni 4000
1978  exit-vrf
1979 !
1980 router bgp 1234
1981  bgp router-id 192.168.0.1
1982  no bgp default ipv4-unicast
1983  no bgp default ipv6-unicast
1984  coalesce-time 1000
1985  neighbor 192.168.0.2 remote-as 1234
1986  neighbor 192.168.0.3 remote-as 1234
1987  neighbor 192.168.0.254 remote-as external
1988  !
1989  address-family ipv4 unicast
1990   import vrf vrf1
1991   neighbor 192.168.0.254 activate
1992  exit-address-family
1993  !
1994  address-family ipv6 unicast
1995   import vrf vrf1
1996   neighbor 192.168.0.254 activate
1997  exit-address-family
1998  !
1999  address-family l2vpn evpn
2000   neighbor 192.168.0.1 activate
2001   neighbor 192.168.0.2 activate
2002   neighbor 192.168.0.254 activate
2003   advertise-all-vni
2004  exit-address-family
2005 !
2006 router bgp 1234 vrf vrf1
2007 !
2008  address-family ipv4 unicast
2009   redistribute connected
2010  exit-address-family
2011  !
2012  address-family ipv6 unicast
2013   redistribute connected
2014  exit-address-family
2015  !
2016  address-family l2vpn evpn
2017   default-originate ipv4
2018   default-originate ipv6
2019  exit-address-family
2020 !
2021 line vty
2022 !
2023 ----
2024
2025 * bgp router
2026
2027 frr.conf
2028 ----
2029 ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24
2030 ip prefix-list NO32 seq 20 deny any
2031 !
2032 router bgp 25253
2033  bgp router-id 192.168.0.254
2034  bgp bestpath as-path multipath-relax
2035  neighbor 192.168.0.1 remote-as external
2036  neighbor 192.168.0.1 capability extended-nexthop
2037  !
2038  address-family ipv4 unicast
2039   neighbor 192.168.0.1 default-originate
2040   neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
2041  exit-address-family
2042  !
2043  address-family ipv6 unicast
2044   neighbor 192.168.0.1 default-originate
2045   neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn
2046  exit-address-family
2047  !
2048 !
2049 ---
2050
2051 Route Reflectors
2052 ^^^^^^^^^^^^^^^^
2053 If you have a lot of proxmox nodes, or multiple proxmox clusters, you may want
2054 to avoid that all node peers with each others nodes.
2055 For this, you can create dedicated route reflectors (RR) servers. As a RR is a
2056 single point of failure, a minimum of two servers acting as an RR is highly
2057 recommended for redundancy.
2058
2059 Below is an example of configuration with 'frr', with `rrserver1
2060 (192.168.0.200)' and `rrserver2 (192.168.0.201)`.
2061
2062 rrserver1
2063 ----
2064 router bgp 1234
2065   bgp router-id 192.168.0.200
2066   bgp cluster-id 1.1.1.1  #cluster-id must be the same on each route reflector
2067   bgp log-neighbor-changes
2068   no bgp default ipv4-unicast
2069   no bgp default ipv6-unicast
2070   neighbor fabric peer-group
2071   neighbor fabric remote-as 1234
2072   neighbor fabric capability extended-nexthop
2073   neighbor fabric update-source 192.168.0.200
2074   bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range
2075   !
2076   address-family l2vpn evpn
2077    neighbor fabric activate
2078    neighbor fabric route-reflector-client
2079    neighbor fabric allowas-in
2080   exit-address-family
2081   !
2082   exit
2083 !
2084 ---
2085
2086 rrserver2
2087 ----
2088 router bgp 1234
2089   bgp router-id 192.168.0.201
2090   bgp cluster-id 1.1.1.1
2091   bgp log-neighbor-changes
2092   no bgp default ipv4-unicast
2093   no bgp default ipv6-unicast
2094   neighbor fabric peer-group
2095   neighbor fabric remote-as 1234
2096   neighbor fabric capability extended-nexthop
2097   neighbor fabric update-source 192.168.0.201
2098   bgp listen range 192.168.0.0/24 peer-group fabric
2099   !
2100   address-family l2vpn evpn
2101    neighbor fabric activate
2102    neighbor fabric route-reflector-client
2103    neighbor fabric allowas-in
2104   exit-address-family
2105   !
2106   exit
2107 !
2108 ---
2109
2110 proxmoxnode(s)
2111 ----
2112 router bgp 1234
2113  bgp router-id 192.168.0.x
2114  no bgp default ipv4-unicast
2115  no bgp default ipv6-unicast
2116  coalesce-time 1000
2117  neighbor 192.168.0.200 remote-as 1234
2118  neighbor 192.168.0.201 remote-as 1234
2119  !
2120  address-family l2vpn evpn
2121   neighbor 192.168.0.200 activate
2122   neighbor 192.168.0.201 activate
2123   advertise-all-vni
2124  exit-address-family
2125 !
2126 ----