vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  21
  22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  23
  24 multicast mode
  25 ^^^^^^^^^^^^^^
  26
  27 This scenario relies in head end replication, meaning that end host in case
  28 of not having any entry for the destination MAC address will send out an ARP
  29 to other devices / VTEPs in the VXLAN network.
  30 This is done by sending the request to the VXLAN multicast group,
  31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  32
  33
  34 * node1
  35
  36 ----
  37 auto eno1
  38 iface eno1 inet manual
  39
  40 auto vmbr0
  41 iface vmbr0 inet static
  42         address 192.168.0.1
  43         netmask  255.255.255.0
  44         bridge_ports eno1
  45         bridge_stp off
  46         bridge_fd 0
  47
  48 auto vxlan2
  49 iface vxlan2 inet manual
  50         vxlan-svcnodeip 225.20.1.1
  51         vxlan-physdev eno1
  52
  53 auto vmbr2
  54 iface vmbr2 inet manual
  55         bridge_ports vxlan2
  56         bridge_stp off
  57         bridge_fd 0
  58
  59 auto vxlan3
  60 iface vxlan3 inet manual
  61         vxlan-svcnodeip 225.20.1.1
  62         vxlan-physdev eno1
  63
  64 auto vmbr3
  65 iface vmbr3 inet manual
  66         bridge_ports vxlan3
  67         bridge_stp off
  68         bridge_fd 0
  69 ----
  70
  71
  72 * node2
  73
  74 ----
  75 auto eno1
  76 iface eno1 inet manual
  77
  78 auto vmbr0
  79 iface vmbr0 inet static
  80         address 192.168.0.2
  81         netmask  255.255.255.0
  82         bridge_ports eno1
  83         bridge_stp off
  84         bridge_fd 0
  85
  86 auto vxlan2
  87 iface vxlan2 inet manual
  88         vxlan-svcnodeip 225.20.1.1
  89         vxlan-physdev eno1
  90
  91 auto vmbr2
  92 iface vmbr2 inet manual
  93         bridge_ports vxlan2
  94         bridge_stp off
  95         bridge_fd 0
  96
  97
  98 auto vxlan3
  99 iface vxlan3 inet manual
 100         vxlan-svcnodeip 225.20.1.1
 101         vxlan-physdev eno1
 102
 103 auto vmbr3
 104 iface vmbr3 inet manual
 105         bridge_ports vxlan3
 106         bridge_stp off
 107         bridge_fd 0
 108 ----
 109
 110
 111 * node3
 112
 113 ----
 114 auto eno1
 115 iface eno1 inet manual
 116
 117 auto vmbr0
 118 iface vmbr0 inet static
 119         address 192.168.0.3
 120         netmask  255.255.255.0
 121         bridge_ports eno1
 122         bridge_stp off
 123         bridge_fd 0
 124
 125 auto vxlan2
 126 iface vxlan2 inet manual
 127         vxlan-svcnodeip 225.20.1.1
 128         vxlan-physdev eno1
 129
 130 auto vmbr2
 131 iface vmbr2 inet manual
 132         bridge_ports vxlan2
 133         bridge_stp off
 134         bridge_fd 0
 135
 136
 137 auto vxlan3
 138 iface vxlan3 inet manual
 139         vxlan-svcnodeip 225.20.1.1
 140         vxlan-physdev eno1
 141
 142 auto vmbr3
 143 iface vmbr3 inet manual
 144         bridge_ports vxlan3
 145         bridge_stp off
 146         bridge_fd 0
 147 ----
 148
 149
 150 unicast mode
 151 ^^^^^^^^^^^^
 152
 153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 154 The VXLAN is defined without a remote multicast group.
 155 Instead, all the remote VTEPs are associated with the all-zero address:
 156 a BUM frame will be duplicated to all these destinations.
 157 The VXLAN device will still learn remote addresses automatically using source-address learning.
 158
 159 * node1
 160
 161 ----
 162 auto eno1
 163 iface eno1 inet manual
 164
 165 auto vmbr0
 166 iface vmbr0 inet static
 167         address 192.168.0.1
 168         netmask  255.255.255.0
 169         bridge_ports eno1
 170         bridge_stp off
 171         bridge_fd 0
 172
 173
 174 auto vxlan2
 175 iface vxlan2 inet manual
 176         vxlan_remoteip 192.168.0.2
 177         vxlan_remoteip 192.168.0.3
 178
 179
 180 auto vmbr2
 181 iface vmbr2 inet manual
 182         bridge_ports vxlan2
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan3
 188 iface vxlan2 inet manual
 189         vxlan_remoteip 192.168.0.2
 190         vxlan_remoteip 192.168.0.3
 191
 192
 193 auto vmbr3
 194 iface vmbr3 inet manual
 195         bridge_ports vxlan3
 196         bridge_stp off
 197         bridge_fd 0
 198 ----
 199
 200
 201 * node2
 202
 203 ----
 204 auto eno1
 205 iface eno1 inet manual
 206
 207 auto vmbr0
 208 iface vmbr0 inet static
 209         address 192.168.0.2
 210         netmask  255.255.255.0
 211         bridge_ports eno1
 212         bridge_stp off
 213         bridge_fd 0
 214
 215 auto vxlan2
 216 iface vxlan2 inet manual
 217         vxlan_remoteip 192.168.0.1
 218         vxlan_remoteip 192.168.0.3
 219
 220
 221
 222 auto vmbr2
 223 iface vmbr2 inet manual
 224         bridge_ports vxlan2
 225         bridge_stp off
 226         bridge_fd 0
 227
 228 auto vxlan3
 229 iface vxlan2 inet manual
 230         vxlan_remoteip 192.168.0.1
 231         vxlan_remoteip 192.168.0.3
 232
 233
 234 auto vmbr3
 235 iface vmbr3 inet manual
 236         bridge_ports vxlan3
 237         bridge_stp off
 238         bridge_fd 0
 239 ----
 240
 241
 242 * node3
 243
 244 ----
 245 auto eno1
 246 iface eno1 inet manual
 247
 248 auto vmbr0
 249 iface vmbr0 inet static
 250         address 192.168.0.3
 251         netmask  255.255.255.0
 252         bridge_ports eno1
 253         bridge_stp off
 254         bridge_fd 0
 255
 256 auto vxlan2
 257 iface vxlan2 inet manual
 258         vxlan_remoteip 192.168.0.2
 259         vxlan_remoteip 192.168.0.3
 260
 261
 262
 263 auto vmbr2
 264 iface vmbr2 inet manual
 265         bridge_ports vxlan2
 266         bridge_stp off
 267         bridge_fd 0
 268
 269 auto vxlan3
 270 iface vxlan2 inet manual
 271         vxlan_remoteip 192.168.0.2
 272         vxlan_remoteip 192.168.0.3
 273
 274
 275 auto vmbr3
 276 iface vmbr3 inet manual
 277         bridge_ports vxlan3
 278         bridge_stp off
 279         bridge_fd 0
 280 ----
 281
 282
 283 bgp-evpn
 284 ^^^^^^^^
 285
 286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 288
 289 The control plane used here is FRR, a bgp routing software.
 290 Each node in the proxmox cluster peer with each others nodes.
 291 For bigger networks, or multiple proxmox clusters,
 292 it's possible to use external bgp route reflector servers.
 293
 294 * node1
 295
 296 ----
 297 auto eno1
 298 iface eno1 inet manual
 299
 300 auto vmbr0
 301 iface vmbr0 inet static
 302         address 192.168.0.1
 303         netmask 255.255.255.0
 304         bridge_ports eno1
 305         bridge_stp off
 306         bridge_fd 0
 307
 308 auto vxlan2
 309 iface vxlan2 inet manual
 310         vxlan-local-tunnelip 192.168.0.1
 311         bridge-learning off
 312         bridge-arp-nd-suppress on
 313         bridge-unicast-flood off
 314         bridge-multicast-flood off
 315
 316
 317 auto vmbr2
 318 iface vmbr2 inet manual
 319         bridge_ports vxlan2
 320         bridge_stp off
 321         bridge_fd 0
 322
 323
 324 auto vxlan3
 325 iface vxlan3 inet manual
 326         vxlan-local-tunnelip 192.168.0.1
 327         bridge-learning off
 328         bridge-arp-nd-suppress on
 329         bridge-unicast-flood off
 330         bridge-multicast-flood off
 331
 332
 333 auto vmbr3
 334 iface vmbr3 inet manual
 335         bridge_ports vxlan3
 336         bridge_stp off
 337         bridge_fd 0
 338 ----
 339
 340
 341 /etc/frr/frr.conf
 342
 343 ----
 344 router bgp 1234
 345  no bgp default ipv4-unicast
 346  coalesce-time 1000
 347  neighbor 192.168.0.2 remote-as 1234
 348  neighbor 192.168.0.3 remote-as 1234
 349  !
 350  address-family l2vpn evpn
 351   neighbor 192.168.0.2 activate
 352   neighbor 192.168.0.3 activate
 353   advertise-all-vni
 354  exit-address-family
 355 !
 356 line vty
 357 !
 358 ----
 359
 360
 361 * node2
 362
 363 ----
 364 auto eno1
 365 iface eno1 inet manual
 366
 367 auto vmbr0
 368 iface vmbr0 inet static
 369         address 192.168.0.2
 370         netmask 255.255.255.0
 371         bridge_ports eno1
 372         bridge_stp off
 373         bridge_fd 0
 374
 375 auto vxlan2
 376 iface vxlan2 inet manual
 377         vxlan-local-tunnelip 192.168.0.2
 378         bridge-learning off
 379         bridge-arp-nd-suppress on
 380         bridge-unicast-flood off
 381         bridge-multicast-flood off
 382
 383
 384 auto vmbr2
 385 iface vmbr2 inet manual
 386         bridge_ports vxlan2
 387         bridge_stp off
 388         bridge_fd 0
 389
 390 auto vxlan3
 391 iface vxlan3 inet manual
 392         vxlan-local-tunnelip 192.168.0.2
 393         bridge-learning off
 394         bridge-arp-nd-suppress on
 395         bridge-unicast-flood off
 396         bridge-multicast-flood off
 397
 398
 399 auto vmbr3
 400 iface vmbr3 inet manual
 401         bridge_ports vxlan3
 402         bridge_stp off
 403         bridge_fd 0
 404 ----
 405
 406
 407 /etc/frr/frr.conf
 408
 409 ----
 410 router bgp 1234
 411  no bgp default ipv4-unicast
 412  coalesce-time 1000
 413  neighbor 192.168.0.1 remote-as 1234
 414  neighbor 192.168.0.3 remote-as 1234
 415  !
 416  address-family l2vpn evpn
 417   neighbor 192.168.0.1 activate
 418   neighbor 192.168.0.3 activate
 419   advertise-all-vni
 420  exit-address-family
 421 !
 422 line vty
 423 !
 424 ----
 425
 426
 427 * node3
 428
 429 ----
 430 auto eno1
 431 iface eno1 inet manual
 432
 433 auto vmbr0
 434 iface vmbr0 inet static
 435         address 192.168.0.2
 436         netmask 255.255.255.0
 437         bridge_ports eno1
 438         bridge_stp off
 439         bridge_fd 0
 440
 441 auto vxlan2
 442 iface vxlan2 inet manual
 443         vxlan-local-tunnelip 192.168.0.3
 444         bridge-learning off
 445         bridge-arp-nd-suppress on
 446         bridge-unicast-flood off
 447         bridge-multicast-flood off
 448
 449
 450 auto vmbr2
 451 iface vmbr2 inet manual
 452         bridge_ports vxlan2
 453         bridge_stp off
 454         bridge_fd 0
 455
 456 auto vxlan3
 457 iface vxlan3 inet manual
 458         vxlan-local-tunnelip 192.168.0.3
 459         bridge-learning off
 460         bridge-arp-nd-suppress on
 461         bridge-unicast-flood off
 462         bridge-multicast-flood off
 463
 464
 465 auto vmbr3
 466 iface vmbr3 inet manual
 467         bridge_ports vxlan3
 468         bridge_stp off
 469         bridge_fd 0
 470 ----
 471
 472
 473 /etc/frr/frr.conf
 474
 475
 476 ----
 477 router bgp 1234
 478  no bgp default ipv4-unicast
 479  coalesce-time 1000
 480  neighbor 192.168.0.1 remote-as 1234
 481  neighbor 192.168.0.2 remote-as 1234
 482  !
 483  address-family l2vpn evpn
 484   neighbor 192.168.0.1 activate
 485   neighbor 192.168.0.2 activate
 486   advertise-all-vni
 487  exit-address-family
 488 !
 489 line vty
 490 !
 491 ----
 492
 493 VXLAN layer3 routing with anycast gateway
 494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 495
 496 With this need, each vmbr bridge will be the gateway for the vm.
 497 Same vmbr on different node, will have same ip address and same mac address,
 498 to have working vm live migration and no network disruption.
 499
 500 VXLAN layer3 routing only work with FRR and non-aware bridge.
 501 (vlan aware bridge support is buggy currently).
 502
 503 asymmetric model
 504 ^^^^^^^^^^^^^^^^
 505
 506 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 507
 508 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 509 but only bridging on the egress.
 510 This results in bi-directional VXLAN traffic traveling on different VNIs
 511 in each direction (always the destination VNI) across the routed infrastructure.
 512
 513 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 514
 515
 516 sysctl.conf tuning
 517
 518 ----
 519 #enable routing
 520 net.ipv4.ip_forward=1
 521 net.ipv6.conf.all.forwarding=1
 522 ----
 523
 524 * node1
 525
 526 ----
 527 auto eno1
 528 iface eno1 inet manual
 529
 530 auto vmbr0
 531 iface vmbr0 inet static
 532          address 192.168.0.1
 533          netmask  255.255.255.0
 534          bridge_ports eno1
 535          bridge_stp off
 536          bridge_fd 0
 537
 538 auto vxlan2
 539 iface vxlan2 inet manual
 540         vxlan-local-tunnelip 192.168.0.1
 541         bridge-learning off
 542         bridge-arp-nd-suppress on
 543         bridge-unicast-flood off
 544         bridge-multicast-flood off
 545
 546
 547 auto vmbr2
 548 iface vmbr2 inet static
 549         address 10.0.2.254
 550         netmask 255.255.255.0
 551         hwaddress 44:39:39:FF:40:94
 552         bridge_ports vxlan2
 553         bridge_stp off
 554         bridge_fd 0
 555
 556
 557 auto vxlan3
 558 iface vxlan3 inet manual
 559         vxlan-local-tunnelip 192.168.0.1
 560         bridge-learning off
 561         bridge-arp-nd-suppress on
 562         bridge-unicast-flood off
 563         bridge-multicast-flood off
 564
 565
 566 auto vmbr3
 567 iface vmbr3 inet static
 568         address 10.0.3.254
 569         netmask 255.255.255.0
 570         hwaddress 44:39:39:FF:40:94
 571         bridge_ports vxlan3
 572         bridge_stp off
 573         bridge_fd 0
 574 ----
 575
 576
 577 frr.conf
 578
 579 ----
 580 router bgp 1234
 581  bgp router-id 192.168.0.1
 582  no bgp default ipv4-unicast
 583  coalesce-time 1000
 584  neighbor 192.168.0.2 remote-as 1234
 585  neighbor 192.168.0.3 remote-as 1234
 586  !
 587  address-family l2vpn evpn
 588   neighbor 192.168.0.2 activate
 589   neighbor 192.168.0.3 activate
 590   advertise-all-vni
 591  exit-address-family
 592 !
 593 line vty
 594 !
 595 ----
 596
 597
 598 * node2
 599
 600 ----
 601 auto eno1
 602 iface eno1 inet manual
 603
 604 auto vmbr0
 605 iface vmbr0 inet static
 606          address 192.168.0.2
 607          netmask  255.255.255.0
 608          bridge_ports eno1
 609          bridge_stp off
 610          bridge_fd 0
 611
 612 auto vxlan2
 613 iface vxlan2 inet manual
 614         vxlan-local-tunnelip 192.168.0.2
 615         bridge-learning off
 616         bridge-arp-nd-suppress on
 617         bridge-unicast-flood off
 618         bridge-multicast-flood off
 619
 620
 621 auto vmbr2
 622 iface vmbr2 inet static
 623         address 10.0.2.254
 624         netmask 255.255.255.0
 625         hwaddress 44:39:39:FF:40:94
 626         bridge_ports vxlan2
 627         bridge_stp off
 628         bridge_fd 0
 629
 630
 631 auto vxlan3
 632 iface vxlan3 inet manual
 633         vxlan-local-tunnelip 192.168.0.2
 634         bridge-learning off
 635         bridge-arp-nd-suppress on
 636         bridge-unicast-flood off
 637         bridge-multicast-flood off
 638
 639
 640 auto vmbr3
 641 iface vmbr3 inet static
 642         address 10.0.3.254
 643         netmask 255.255.255.0
 644         hwaddress 44:39:39:FF:40:94
 645         bridge_ports vxlan3
 646         bridge_stp off
 647         bridge_fd 0
 648 ----
 649
 650
 651 frr.conf
 652
 653 ----
 654 router bgp 1234
 655  bgp router-id 192.168.0.2
 656  no bgp default ipv4-unicast
 657  coalesce-time 1000
 658  neighbor 192.168.0.1 remote-as 1234
 659  neighbor 192.168.0.3 remote-as 1234
 660  !
 661  address-family l2vpn evpn
 662   neighbor 192.168.0.1 activate
 663   neighbor 192.168.0.3 activate
 664   advertise-all-vni
 665  exit-address-family
 666 !
 667 line vty
 668 !
 669 ----
 670
 671
 672 * node3
 673
 674 ----
 675 auto eno1
 676 iface eno1 inet manual
 677
 678 auto vmbr0
 679 iface vmbr0 inet static
 680          address 192.168.0.3
 681          netmask  255.255.255.0
 682          bridge_ports eno1
 683          bridge_stp off
 684          bridge_fd 0
 685
 686 auto vxlan2
 687 iface vxlan2 inet manual
 688         vxlan-local-tunnelip 192.168.0.3
 689         bridge-learning off
 690         bridge-arp-nd-suppress on
 691         bridge-unicast-flood off
 692         bridge-multicast-flood off
 693
 694
 695 auto vmbr2
 696 iface vmbr2 inet static
 697         address 10.0.2.254
 698         netmask 255.255.255.0
 699         hwaddress 44:39:39:FF:40:94
 700         bridge_ports vxlan2
 701         bridge_stp off
 702         bridge_fd 0
 703
 704
 705 auto vxlan3
 706 iface vxlan3 inet manual
 707         vxlan-local-tunnelip 192.168.0.3
 708         bridge-learning off
 709         bridge-arp-nd-suppress on
 710         bridge-unicast-flood off
 711         bridge-multicast-flood off
 712
 713
 714 auto vmbr3
 715 iface vmbr3 inet static
 716         address 10.0.3.254
 717         netmask 255.255.255.0
 718         hwaddress 44:39:39:FF:40:94
 719         bridge_ports vxlan3
 720         bridge_stp off
 721         bridge_fd 0
 722 ----
 723
 724
 725 frr.conf
 726
 727 ----
 728 router bgp 1234
 729  bgp router-id 192.168.0.3
 730  no bgp default ipv4-unicast
 731  coalesce-time 1000
 732  neighbor 192.168.0.1 remote-as 1234
 733  neighbor 192.168.0.2 remote-as 1234
 734  !
 735  address-family l2vpn evpn
 736   neighbor 192.168.0.1 activate
 737   neighbor 192.168.0.2 activate
 738   advertise-all-vni
 739  exit-address-family
 740 !
 741 line vty
 742 !
 743 ----
 744
 745
 746 symmetric model
 747 ^^^^^^^^^^^^^^^
 748
 749 With this model, you don't need to have all vxlan on all nodes.
 750 This model will also be needed to route traffic to an external router.
 751
 752 The symmetric model routes and bridges on both the ingress and the egress leafs.
 753 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 754 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 755 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 756 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 757
 758 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 759
 760 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 761
 762 sysctl.conf tuning
 763
 764 ----
 765 #enable routing
 766 net.ipv4.ip_forward=1
 767 net.ipv6.conf.all.forwarding=1
 768 #disable reverse path filtering
 769 net.ipv4.conf.default.rp_filter=0
 770 net.ipv4.conf.all.rp_filter=0
 771 ----
 772
 773 * node1
 774
 775 ----
 776 auto vrf1
 777 iface vrf1
 778     vrf-table auto
 779
 780 auto eno1
 781 iface eno1 inet manual
 782
 783 auto vmbr0
 784 iface vmbr0 inet static
 785          address 192.168.0.1
 786          netmask  255.255.255.0
 787          bridge_ports eno1
 788          bridge_stp off
 789          bridge_fd 0
 790
 791 auto vxlan2
 792 iface vxlan2 inet manual
 793         vxlan-local-tunnelip 192.168.0.1
 794         bridge-learning off
 795         bridge-arp-nd-suppress on
 796         bridge-unicast-flood off
 797         bridge-multicast-flood off
 798
 799 auto vmbr2
 800 iface vmbr2 inet static
 801         bridge_ports vxlan2
 802         bridge_stp off
 803         bridge_fd 0
 804         address 10.0.2.254
 805         netmask 255.255.255.0
 806         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 807         vrf vrf1
 808
 809 auto vxlan3
 810 iface vxlan3 inet manual
 811         vxlan-local-tunnelip 192.168.0.1
 812         bridge-learning off
 813         bridge-arp-nd-suppress on
 814         bridge-unicast-flood off
 815         bridge-multicast-flood off
 816
 817 auto vmbr3
 818 iface vmbr3 inet static
 819         bridge_ports vxlan3
 820         bridge_stp off
 821         bridge_fd 0
 822         address 10.0.3.254
 823         netmask 255.255.255.0
 824         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 825         vrf vrf1
 826
 827 #interconnect vxlan-vfr l3vni
 828 auto vxlan4000
 829 iface vxlan4000 inet manual
 830         vxlan-local-tunnelip 192.168.0.1
 831         bridge-learning off
 832         bridge-arp-nd-suppress on
 833         bridge-unicast-flood off
 834         bridge-multicast-flood off
 835
 836
 837 auto vmbr4000
 838 iface vmbr4000 inet manual
 839         bridge_ports vxlan4000
 840         bridge_stp off
 841         bridge_fd 0
 842         hwaddress 44:39:39:FF:40:90  #must be different on each node
 843         vrf vrf1
 844 ----
 845
 846 frr.conf
 847
 848 ----
 849 vrf vrf1
 850  vni 4000
 851 !
 852 router bgp 1234
 853  bgp router-id 192.168.0.1
 854  no bgp default ipv4-unicast
 855  coalesce-time 1000
 856  neighbor 192.168.0.2 remote-as 1234
 857  neighbor 192.168.0.3 remote-as 1234
 858  !
 859  address-family l2vpn evpn
 860   neighbor 192.168.0.2 activate
 861   neighbor 192.168.0.3 activate
 862   advertise-all-vni
 863  exit-address-family
 864 !
 865 router bgp 1234 vrf vrf1
 866 !
 867  bgp router-id 192.168.0.1
 868  !
 869  address-family ipv4 unicast
 870   redistribute connected
 871  exit-address-family
 872  !
 873  address-family l2vpn evpn
 874   advertise ipv4 unicast
 875  exit-address-family
 876 !
 877 line vty
 878 !
 879 ----
 880
 881
 882 * node2
 883
 884 ----
 885 auto vrf1
 886 iface vrf1
 887     vrf-table auto
 888
 889 auto eno1
 890 iface eno1 inet manual
 891
 892 auto vmbr0
 893 iface vmbr0 inet static
 894          address 192.168.0.2
 895          netmask  255.255.255.0
 896          bridge_ports eno1
 897          bridge_stp off
 898          bridge_fd 0
 899
 900 auto vxlan2
 901 iface vxlan2 inet manual
 902         vxlan-local-tunnelip 192.168.0.2
 903         bridge-learning off
 904         bridge-arp-nd-suppress on
 905         bridge-unicast-flood off
 906         bridge-multicast-flood off
 907
 908 auto vmbr2
 909 iface vmbr2 inet static
 910         bridge_ports vxlan2
 911         bridge_stp off
 912         bridge_fd 0
 913         address 10.0.2.254
 914         netmask 255.255.255.0
 915         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 916         vrf vrf1
 917
 918 auto vxlan3
 919 iface vxlan3 inet manual
 920         vxlan-local-tunnelip 192.168.0.2
 921         bridge-learning off
 922         bridge-arp-nd-suppress on
 923         bridge-unicast-flood off
 924         bridge-multicast-flood off
 925
 926 auto vmbr3
 927 iface vmbr3 inet static
 928         bridge_ports vxlan3
 929         bridge_stp off
 930         bridge_fd 0
 931         address 10.0.3.254
 932         netmask 255.255.255.0
 933         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 934         vrf vrf1
 935
 936 #interconnect vxlan-vfr l3vni
 937 auto vxlan4000
 938 iface vxlan4000 inet manual
 939         vxlan-local-tunnelip 192.168.0.2
 940         bridge-learning off
 941         bridge-arp-nd-suppress on
 942         bridge-unicast-flood off
 943         bridge-multicast-flood off
 944
 945
 946 auto vmbr4000
 947 iface vmbr4000 inet manual
 948         bridge_ports vxlan4000
 949         bridge_stp off
 950         bridge_fd 0
 951         hwaddress 44:39:39:FF:40:91  #must be different on each node
 952         vrf vrf1
 953 ----
 954
 955
 956 frr.conf
 957
 958 ----
 959 vrf vrf1
 960  vni 4000
 961 !
 962 router bgp 1234
 963  bgp router-id 192.168.0.2
 964  no bgp default ipv4-unicast
 965  coalesce-time 1000
 966  neighbor 192.168.0.1 remote-as 1234
 967  neighbor 192.168.0.3 remote-as 1234
 968  !
 969  address-family l2vpn evpn
 970   neighbor 192.168.0.1 activate
 971   neighbor 192.168.0.3 activate
 972   advertise-all-vni
 973  exit-address-family
 974 !
 975 router bgp 1234 vrf vrf1
 976 !
 977  bgp router-id 192.168.0.2
 978  !
 979  address-family ipv4 unicast
 980   redistribute connected
 981  exit-address-family
 982  !
 983  address-family l2vpn evpn
 984   advertise ipv4 unicast
 985  exit-address-family
 986 !
 987 line vty
 988 !
 989 ----
 990
 991
 992 * node3
 993
 994 ----
 995 auto vrf1
 996 iface vrf1
 997     vrf-table auto
 998
 999 auto eno1
1000 iface eno1 inet manual
1001
1002 auto vmbr0
1003 iface vmbr0 inet static
1004          address 192.168.0.3
1005          netmask  255.255.255.0
1006          bridge_ports eno1
1007          bridge_stp off
1008          bridge_fd 0
1009
1010 auto vxlan2
1011 iface vxlan2 inet manual
1012         vxlan-local-tunnelip 192.168.0.3
1013         bridge-learning off
1014         bridge-arp-nd-suppress on
1015         bridge-unicast-flood off
1016         bridge-multicast-flood off
1017
1018 auto vmbr2
1019 iface vmbr2 inet static
1020         bridge_ports vxlan2
1021         bridge_stp off
1022         bridge_fd 0
1023         address 10.0.2.254
1024         netmask 255.255.255.0
1025         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1026         vrf vrf1
1027
1028 auto vxlan3
1029 iface vxlan3 inet manual
1030         vxlan-local-tunnelip 192.168.0.3
1031         bridge-learning off
1032         bridge-arp-nd-suppress on
1033         bridge-unicast-flood off
1034         bridge-multicast-flood off
1035
1036 auto vmbr3
1037 iface vmbr3 inet static
1038         bridge_ports vxlan3
1039         bridge_stp off
1040         bridge_fd 0
1041         address 10.0.3.254
1042         netmask 255.255.255.0
1043         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1044         vrf vrf1
1045
1046 #interconnect vxlan-vfr l3vni
1047 auto vxlan4000
1048 iface vxlan4000 inet manual
1049         vxlan-local-tunnelip 192.168.0.3
1050         bridge-learning off
1051         bridge-arp-nd-suppress on
1052         bridge-unicast-flood off
1053         bridge-multicast-flood off
1054
1055
1056 auto vmbr4000
1057 iface vmbr4000 inet manual
1058         bridge_ports vxlan4000
1059         bridge_stp off
1060         bridge_fd 0
1061         hwaddress 44:39:39:FF:40:92  #must be different on each node
1062         vrf vrf1
1063 ----
1064
1065
1066 frr.conf
1067
1068 ----
1069 vrf vrf1
1070  vni 4000
1071 !
1072 router bgp 1234
1073  bgp router-id 192.168.0.3
1074  no bgp default ipv4-unicast
1075  coalesce-time 1000
1076  neighbor 192.168.0.1 remote-as 1234
1077  neighbor 192.168.0.2 remote-as 1234
1078  !
1079  address-family l2vpn evpn
1080   neighbor 192.168.0.1 activate
1081   neighbor 192.168.0.2 activate
1082   advertise-all-vni
1083  exit-address-family
1084 !
1085 router bgp 1234 vrf vrf1
1086 !
1087  bgp router-id 192.168.0.3
1088  !
1089  address-family ipv4 unicast
1090   redistribute connected
1091  exit-address-family
1092  !
1093  address-family l2vpn evpn
1094   advertise ipv4 unicast
1095  exit-address-family
1096 !
1097 line vty
1098 !
1099 ----
1100
1101 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1103 Routing to outside need the symmetric model.
1104
1105 1 gateway node
1106 ^^^^^^^^^^^^^^
1107 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1108 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1109 and announce this default gw to other proxmox nodes.
1110
1111
1112 *node1
1113
1114 ----
1115 auto vrf1
1116 iface vrf1
1117     vrf-table auto
1118
1119 auto eno1
1120 iface eno1 inet manual
1121
1122 auto vmbr0
1123 iface vmbr0 inet static
1124          address 192.168.0.1
1125          netmask  255.255.255.0
1126          bridge_ports eno1
1127          bridge_stp off
1128          bridge_fd 0
1129
1130 auto eno2
1131 iface eno2
1132         address 172.16.0.1
1133         netmask 255.255.255.0
1134         vrf vrf1
1135         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1136         #if you have multiple external routers, you can use ecmp balancing
1137         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1138
1139 auto vxlan2
1140 iface vxlan2 inet manual
1141         vxlan-local-tunnelip 192.168.0.1
1142         bridge-learning off
1143         bridge-arp-nd-suppress on
1144         bridge-unicast-flood off
1145         bridge-multicast-flood off
1146
1147 auto vmbr2
1148 iface vmbr2 inet static
1149         bridge_ports vxlan2
1150         bridge_stp off
1151         bridge_fd 0
1152         address 10.0.2.254
1153         netmask 255.255.255.0
1154         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1155         vrf vrf1
1156
1157 auto vxlan3
1158 iface vxlan3 inet manual
1159         vxlan-local-tunnelip 192.168.0.1
1160         bridge-learning off
1161         bridge-arp-nd-suppress on
1162         bridge-unicast-flood off
1163         bridge-multicast-flood off
1164
1165 auto vmbr3
1166 iface vmbr3 inet static
1167         bridge_ports vxlan3
1168         bridge_stp off
1169         bridge_fd 0
1170         address 10.0.3.254
1171         netmask 255.255.255.0
1172         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1173         vrf vrf1
1174
1175 #interconnect vxlan-vfr l3vni
1176 auto vxlan4000
1177 iface vxlan4000 inet manual
1178         vxlan-local-tunnelip 192.168.0.1
1179         bridge-learning off
1180         bridge-arp-nd-suppress on
1181         bridge-unicast-flood off
1182         bridge-multicast-flood off
1183
1184 auto vmbr4000
1185 iface vmbr4000 inet manual
1186         bridge_ports vxlan4000
1187         bridge_stp off
1188         bridge_fd 0
1189         hwaddress 44:39:39:FF:40:90  #must be different on each node
1190         vrf vrf1
1191 ----
1192
1193
1194 frr.conf
1195
1196 ----
1197 vrf vrf1
1198  vni 4000
1199 !
1200 router bgp 1234
1201  bgp router-id 192.168.0.1
1202  no bgp default ipv4-unicast
1203  coalesce-time 1000
1204  neighbor 192.168.0.2 remote-as 1234
1205  neighbor 192.168.0.3 remote-as 1234
1206  !
1207  address-family l2vpn evpn
1208   neighbor 192.168.0.2 activate
1209   neighbor 192.168.0.3 activate
1210   advertise-all-vni
1211  exit-address-family
1212 !
1213 router bgp 1234 vrf vrf1
1214 !
1215  bgp router-id 172.16.0.1
1216  !
1217  address-family ipv4 unicast
1218   redistribute connected
1219   redistribute kernel !announce your default gw to all nodes
1220  exit-address-family
1221  !
1222  address-family l2vpn evpn
1223   advertise ipv4 unicast
1224  exit-address-family
1225 !
1226 line vty
1227 !
1228 ----
1229
1230
1231 * node2
1232
1233 ----
1234 auto vrf1
1235 iface vrf1
1236     vrf-table auto
1237
1238 auto eno1
1239 iface eno1 inet manual
1240
1241 auto vmbr0
1242 iface vmbr0 inet static
1243          address 192.168.0.2
1244          netmask  255.255.255.0
1245          bridge_ports eno1
1246          bridge_stp off
1247          bridge_fd 0
1248
1249 auto vxlan2
1250 iface vxlan2 inet manual
1251         vxlan-local-tunnelip 192.168.0.2
1252         bridge-learning off
1253         bridge-arp-nd-suppress on
1254         bridge-unicast-flood off
1255         bridge-multicast-flood off
1256
1257 auto vmbr2
1258 iface vmbr2 inet static
1259         bridge_ports vxlan2
1260         bridge_stp off
1261         bridge_fd 0
1262         address 10.0.2.254
1263         netmask 255.255.255.0
1264         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1265         vrf vrf1
1266
1267 auto vxlan3
1268 iface vxlan3 inet manual
1269         vxlan-local-tunnelip 192.168.0.2
1270         bridge-learning off
1271         bridge-arp-nd-suppress on
1272         bridge-unicast-flood off
1273         bridge-multicast-flood off
1274
1275 auto vmbr3
1276 iface vmbr3 inet static
1277         bridge_ports vxlan3
1278         bridge_stp off
1279         bridge_fd 0
1280         address 10.0.3.254
1281         netmask 255.255.255.0
1282         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1283         vrf vrf1
1284
1285 #interconnect vxlan-vfr l3vni
1286 auto vxlan4000
1287 iface vxlan4000 inet manual
1288         vxlan-local-tunnelip 192.168.0.2
1289         bridge-learning off
1290         bridge-arp-nd-suppress on
1291         bridge-unicast-flood off
1292         bridge-multicast-flood off
1293
1294
1295 auto vmbr4000
1296 iface vmbr4000 inet manual
1297         bridge_ports vxlan4000
1298         bridge_stp off
1299         bridge_fd 0
1300         hwaddress 44:39:39:FF:40:91  #must be different on each node
1301         vrf vrf1
1302 ----
1303
1304
1305 frr.conf
1306
1307 ----
1308 vrf vrf1
1309  vni 4000
1310 !
1311 router bgp 1234
1312  bgp router-id 192.168.0.2
1313  no bgp default ipv4-unicast
1314  coalesce-time 1000
1315  neighbor 192.168.0.1 remote-as 1234
1316  neighbor 192.168.0.3 remote-as 1234
1317  !
1318  address-family l2vpn evpn
1319   neighbor 192.168.0.1 activate
1320   neighbor 192.168.0.3 activate
1321   advertise-all-vni
1322  exit-address-family
1323 !
1324 router bgp 1234 vrf vrf1
1325 !
1326  bgp router-id 192.168.0.2
1327  !
1328  address-family ipv4 unicast
1329   redistribute connected
1330  exit-address-family
1331  !
1332  address-family l2vpn evpn
1333   advertise ipv4 unicast
1334  exit-address-family
1335 !
1336 line vty
1337 !
1338 ----
1339
1340
1341 * node3
1342
1343 ----
1344 auto vrf1
1345 iface vrf1
1346     vrf-table auto
1347
1348 auto eno1
1349 iface eno1 inet manual
1350
1351 auto vmbr0
1352 iface vmbr0 inet static
1353          address 192.168.0.3
1354          netmask  255.255.255.0
1355          bridge_ports eno1
1356          bridge_stp off
1357          bridge_fd 0
1358
1359 auto vxlan2
1360 iface vxlan2 inet manual
1361         vxlan-local-tunnelip 192.168.0.3
1362         bridge-learning off
1363         bridge-arp-nd-suppress on
1364         bridge-unicast-flood off
1365         bridge-multicast-flood off
1366
1367 auto vmbr2
1368 iface vmbr2 inet static
1369         bridge_ports vxlan2
1370         bridge_stp off
1371         bridge_fd 0
1372         address 10.0.2.254
1373         netmask 255.255.255.0
1374         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1375         vrf vrf1
1376
1377 auto vxlan3
1378 iface vxlan3 inet manual
1379         vxlan-local-tunnelip 192.168.0.3
1380         bridge-learning off
1381         bridge-arp-nd-suppress on
1382         bridge-unicast-flood off
1383         bridge-multicast-flood off
1384
1385 auto vmbr3
1386 iface vmbr3 inet static
1387         bridge_ports vxlan3
1388         bridge_stp off
1389         bridge_fd 0
1390         address 10.0.3.254
1391         netmask 255.255.255.0
1392         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1393         vrf vrf1
1394
1395 #interconnect vxlan-vfr l3vni
1396 auto vxlan4000
1397 iface vxlan4000 inet manual
1398         vxlan-local-tunnelip 192.168.0.3
1399         bridge-learning off
1400         bridge-arp-nd-suppress on
1401         bridge-unicast-flood off
1402         bridge-multicast-flood off
1403
1404
1405 auto vmbr4000
1406 iface vmbr4000 inet manual
1407         bridge_ports vxlan4000
1408         bridge_stp off
1409         bridge_fd 0
1410         hwaddress 44:39:39:FF:40:92  #must be different on each node
1411         vrf vrf1
1412 ----
1413
1414
1415 frr.conf
1416
1417 ----
1418 vrf vrf1
1419  vni 4000
1420 !
1421 router bgp 1234
1422  bgp router-id 192.168.0.3
1423  no bgp default ipv4-unicast
1424  coalesce-time 1000
1425  neighbor 192.168.0.1 remote-as 1234
1426  neighbor 192.168.0.2 remote-as 1234
1427  !
1428  address-family l2vpn evpn
1429   neighbor 192.168.0.1 activate
1430   neighbor 192.168.0.2 activate
1431   advertise-all-vni
1432  exit-address-family
1433 !
1434 router bgp 1234 vrf vrf1
1435 !
1436  bgp router-id 192.168.0.3
1437  !
1438  address-family ipv4 unicast
1439   redistribute connected
1440  exit-address-family
1441  !
1442  address-family l2vpn evpn
1443   advertise ipv4 unicast
1444  exit-address-family
1445 !
1446 line vty
1447 !
1448 ----
1449
1450 multiple gateway nodes
1451 ^^^^^^^^^^^^^^^^^^^^^^
1452 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1453 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1454 and announce this default gw.
1455 The external router have ecmp routes to all proxmox nodes.(balancing).
1456 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1457 vxlan the packet to final destination.
1458
1459 *node1
1460
1461 ----
1462 auto vrf1
1463 iface vrf1
1464     vrf-table auto
1465
1466 auto eno1
1467 iface eno1 inet manual
1468
1469 auto vmbr0
1470 iface vmbr0 inet static
1471          address 192.168.0.1
1472          netmask  255.255.255.0
1473          bridge_ports eno1
1474          bridge_stp off
1475          bridge_fd 0
1476
1477 auto eno2
1478 iface eno2
1479         address 172.16.0.1
1480         netmask 255.255.255.0
1481         vrf vrf1
1482         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1483         #if you have multiple external routers, you can use ecmp balancing
1484         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1485
1486 auto vxlan2
1487 iface vxlan2 inet manual
1488         vxlan-local-tunnelip 192.168.0.1
1489         bridge-learning off
1490         bridge-arp-nd-suppress on
1491         bridge-unicast-flood off
1492         bridge-multicast-flood off
1493
1494 auto vmbr2
1495 iface vmbr2 inet static
1496         bridge_ports vxlan2
1497         bridge_stp off
1498         bridge_fd 0
1499         address 10.0.2.254
1500         netmask 255.255.255.0
1501         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1502         vrf vrf1
1503
1504 auto vxlan3
1505 iface vxlan3 inet manual
1506         vxlan-local-tunnelip 192.168.0.1
1507         bridge-learning off
1508         bridge-arp-nd-suppress on
1509         bridge-unicast-flood off
1510         bridge-multicast-flood off
1511
1512 auto vmbr3
1513 iface vmbr3 inet static
1514         bridge_ports vxlan3
1515         bridge_stp off
1516         bridge_fd 0
1517         address 10.0.3.254
1518         netmask 255.255.255.0
1519         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1520         vrf vrf1
1521
1522 #interconnect vxlan-vfr l3vni
1523 auto vxlan4000
1524 iface vxlan4000 inet manual
1525         vxlan-local-tunnelip 192.168.0.1
1526         bridge-learning off
1527         bridge-arp-nd-suppress on
1528         bridge-unicast-flood off
1529         bridge-multicast-flood off
1530
1531 auto vmbr4000
1532 iface vmbr4000 inet manual
1533         bridge_ports vxlan4000
1534         bridge_stp off
1535         bridge_fd 0
1536         hwaddress 44:39:39:FF:40:90  #must be different on each node
1537         vrf vrf1
1538 ----
1539
1540
1541 frr.conf
1542
1543 ----
1544 vrf vrf1
1545  vni 4000
1546 !
1547 router bgp 1234
1548  bgp router-id 192.168.0.1
1549  no bgp default ipv4-unicast
1550  coalesce-time 1000
1551  neighbor 192.168.0.2 remote-as 1234
1552  neighbor 192.168.0.3 remote-as 1234
1553  !
1554  address-family l2vpn evpn
1555   neighbor 192.168.0.2 activate
1556   neighbor 192.168.0.3 activate
1557   advertise-all-vni
1558  exit-address-family
1559 !
1560 router bgp 1234 vrf vrf1
1561 !
1562  bgp router-id 172.16.0.1
1563  !
1564  address-family ipv4 unicast
1565   redistribute connected
1566   redistribute kernel !announce your default gw to all nodes
1567  exit-address-family
1568  !
1569  address-family l2vpn evpn
1570   advertise ipv4 unicast
1571  exit-address-family
1572 !
1573 line vty
1574 !
1575 ----
1576
1577
1578 * node2
1579
1580 ----
1581 auto vrf1
1582 iface vrf1
1583     vrf-table auto
1584
1585 auto eno1
1586 iface eno1 inet manual
1587
1588 auto vmbr0
1589 iface vmbr0 inet static
1590          address 192.168.0.2
1591          netmask  255.255.255.0
1592          bridge_ports eno1
1593          bridge_stp off
1594          bridge_fd 0
1595
1596 auto eno2
1597 iface eno2
1598         address 172.16.0.3
1599         netmask 255.255.255.0
1600         vrf vrf1
1601         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1602         #if you have multiple external routers, you can use ecmp balancing
1603         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1604
1605 auto vxlan2
1606 iface vxlan2 inet manual
1607         vxlan-local-tunnelip 192.168.0.2
1608         bridge-learning off
1609         bridge-arp-nd-suppress on
1610         bridge-unicast-flood off
1611         bridge-multicast-flood off
1612
1613 auto vmbr2
1614 iface vmbr2 inet static
1615         bridge_ports vxlan2
1616         bridge_stp off
1617         bridge_fd 0
1618         address 10.0.2.254
1619         netmask 255.255.255.0
1620         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1621         vrf vrf1
1622
1623 auto vxlan3
1624 iface vxlan3 inet manual
1625         vxlan-local-tunnelip 192.168.0.2
1626         bridge-learning off
1627         bridge-arp-nd-suppress on
1628         bridge-unicast-flood off
1629         bridge-multicast-flood off
1630
1631 auto vmbr3
1632 iface vmbr3 inet static
1633         bridge_ports vxlan3
1634         bridge_stp off
1635         bridge_fd 0
1636         address 10.0.3.254
1637         netmask 255.255.255.0
1638         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1639         vrf vrf1
1640
1641 #interconnect vxlan-vfr l3vni
1642 auto vxlan4000
1643 iface vxlan4000 inet manual
1644         vxlan-local-tunnelip 192.168.0.2
1645         bridge-learning off
1646         bridge-arp-nd-suppress on
1647         bridge-unicast-flood off
1648         bridge-multicast-flood off
1649
1650
1651 auto vmbr4000
1652 iface vmbr4000 inet manual
1653         bridge_ports vxlan4000
1654         bridge_stp off
1655         bridge_fd 0
1656         hwaddress 44:39:39:FF:40:91  #must be different on each node
1657         vrf vrf1
1658 ----
1659
1660
1661 frr.conf
1662
1663 ----
1664 vrf vrf1
1665  vni 4000
1666 !
1667 router bgp 1234
1668  bgp router-id 192.168.0.2
1669  no bgp default ipv4-unicast
1670  coalesce-time 1000
1671  neighbor 192.168.0.1 remote-as 1234
1672  neighbor 192.168.0.3 remote-as 1234
1673  !
1674  address-family l2vpn evpn
1675   neighbor 192.168.0.1 activate
1676   neighbor 192.168.0.3 activate
1677   advertise-all-vni
1678  exit-address-family
1679 !
1680 router bgp 1234 vrf vrf1
1681 !
1682  bgp router-id 172.16.0.2
1683  !
1684  address-family ipv4 unicast
1685   redistribute connected
1686   redistribute kernel !announce your default gw to all nodes
1687  exit-address-family
1688  !
1689  address-family l2vpn evpn
1690   advertise ipv4 unicast
1691  exit-address-family
1692 !
1693 line vty
1694 !
1695 ----
1696
1697
1698 * node3
1699
1700 ----
1701 auto vrf1
1702 iface vrf1
1703     vrf-table auto
1704
1705 auto eno1
1706 iface eno1 inet manual
1707
1708 auto vmbr0
1709 iface vmbr0 inet static
1710          address 192.168.0.3
1711          netmask  255.255.255.0
1712          bridge_ports eno1
1713          bridge_stp off
1714          bridge_fd 0
1715
1716 auto eno2
1717 iface eno2
1718         address 172.16.0.3
1719         netmask 255.255.255.0
1720         vrf vrf1
1721         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1722         #if you have multiple external routers, you can use ecmp balancing
1723         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1724
1725 auto vxlan2
1726 iface vxlan2 inet manual
1727         vxlan-local-tunnelip 192.168.0.3
1728         bridge-learning off
1729         bridge-arp-nd-suppress on
1730         bridge-unicast-flood off
1731         bridge-multicast-flood off
1732
1733 auto vmbr2
1734 iface vmbr2 inet static
1735         bridge_ports vxlan2
1736         bridge_stp off
1737         bridge_fd 0
1738         address 10.0.2.254
1739         netmask 255.255.255.0
1740         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1741         vrf vrf1
1742
1743 auto vxlan3
1744 iface vxlan3 inet manual
1745         vxlan-local-tunnelip 192.168.0.3
1746         bridge-learning off
1747         bridge-arp-nd-suppress on
1748         bridge-unicast-flood off
1749         bridge-multicast-flood off
1750
1751 auto vmbr3
1752 iface vmbr3 inet static
1753         bridge_ports vxlan3
1754         bridge_stp off
1755         bridge_fd 0
1756         address 10.0.3.254
1757         netmask 255.255.255.0
1758         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1759         vrf vrf1
1760
1761 #interconnect vxlan-vfr l3vni
1762 auto vxlan4000
1763 iface vxlan4000 inet manual
1764         vxlan-local-tunnelip 192.168.0.3
1765         bridge-learning off
1766         bridge-arp-nd-suppress on
1767         bridge-unicast-flood off
1768         bridge-multicast-flood off
1769
1770
1771 auto vmbr4000
1772 iface vmbr4000 inet manual
1773         bridge_ports vxlan4000
1774         bridge_stp off
1775         bridge_fd 0
1776         hwaddress 44:39:39:FF:40:92  #must be different on each node
1777         vrf vrf1
1778 ----
1779
1780
1781 frr.conf
1782
1783 ----
1784 vrf vrf1
1785  vni 4000
1786 !
1787 router bgp 1234
1788  bgp router-id 192.168.0.3
1789  no bgp default ipv4-unicast
1790  coalesce-time 1000
1791  neighbor 192.168.0.1 remote-as 1234
1792  neighbor 192.168.0.2 remote-as 1234
1793  !
1794  address-family l2vpn evpn
1795   neighbor 192.168.0.1 activate
1796   neighbor 192.168.0.2 activate
1797   advertise-all-vni
1798  exit-address-family
1799 !
1800 router bgp 1234 vrf vrf1
1801 !
1802  bgp router-id 172.16.0.3
1803  !
1804  address-family ipv4 unicast
1805   redistribute connected
1806   redistribute kernel !announce your default gw to all nodes
1807  exit-address-family
1808  !
1809  address-family l2vpn evpn
1810   advertise ipv4 unicast
1811  exit-address-family
1812 !
1813 line vty
1814 !
1815 ----
1816
1817 Note
1818 ^^^^
1819
1820 If your external router don't support ecmp to reach multiple proxmox nodes,
1821 you can setup an HA floating vip on proxmox nodes with vrrp
1822
1823 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1824 Node1 is the primary and failover to node2 in case of failure.
1825
1826
1827 * node1
1828
1829 ----
1830 auto eno2
1831 iface eno2
1832         address 172.16.0.1
1833         netmask 255.255.255.0
1834         vrf vrf1
1835         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1836         vrrp-id 1
1837         vrrp-priority 1
1838         vrrp-virtual-ip 172.16.0.10
1839 ----
1840
1841 * node2
1842
1843 ----
1844 auto eno2
1845 iface eno2
1846         address 172.16.0.2
1847         netmask 255.255.255.0
1848         vrf vrf1
1849         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1850         vrrp-id 1
1851         vrrp-priority 2
1852         vrrp-virtual-ip 172.16.0.10
1853 ----
1854
1855