vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  20 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  21
  22 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  23
  24 multicast mode
  25 ^^^^^^^^^^^^^^
  26
  27 This scenario relies in head end replication, meaning that end host in case
  28 of not having any entry for the destination MAC address will send out an ARP
  29 to other devices / VTEPs in the VXLAN network.
  30 This is done by sending the request to the VXLAN multicast group,
  31 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  32
  33
  34 * node1
  35
  36 ----
  37 auto eno1
  38 iface eno1 inet manual
  39
  40 auto vmbr0
  41 iface vmbr0 inet static
  42         address 192.168.0.1
  43         netmask  255.255.255.0
  44         bridge_ports eno1
  45         bridge_stp off
  46         bridge_fd 0
  47
  48 auto vxlan2
  49 iface vxlan2 inet manual
  50         vxlan-svcnodeip 225.20.1.1
  51         vxlan-physdev eno1
  52
  53 auto vmbr2
  54 iface vmbr2 inet manual
  55         bridge_ports vxlan2
  56         bridge_stp off
  57         bridge_fd 0
  58
  59 auto vxlan3
  60 iface vxlan3 inet manual
  61         vxlan-svcnodeip 225.20.1.1
  62         vxlan-physdev eno1
  63
  64 auto vmbr3
  65 iface vmbr3 inet manual
  66         bridge_ports vxlan3
  67         bridge_stp off
  68         bridge_fd 0
  69 ----
  70
  71
  72 * node2
  73
  74 ----
  75 auto eno1
  76 iface eno1 inet manual
  77
  78 auto vmbr0
  79 iface vmbr0 inet static
  80         address 192.168.0.2
  81         netmask  255.255.255.0
  82         bridge_ports eno1
  83         bridge_stp off
  84         bridge_fd 0
  85
  86 auto vxlan2
  87 iface vxlan2 inet manual
  88         vxlan-svcnodeip 225.20.1.1
  89         vxlan-physdev eno1
  90
  91 auto vmbr2
  92 iface vmbr2 inet manual
  93         bridge_ports vxlan2
  94         bridge_stp off
  95         bridge_fd 0
  96
  97
  98 auto vxlan3
  99 iface vxlan3 inet manual
 100         vxlan-svcnodeip 225.20.1.1
 101         vxlan-physdev eno1
 102
 103 auto vmbr3
 104 iface vmbr3 inet manual
 105         bridge_ports vxlan3
 106         bridge_stp off
 107         bridge_fd 0
 108 ----
 109
 110
 111 * node3
 112
 113 ----
 114 auto eno1
 115 iface eno1 inet manual
 116
 117 auto vmbr0
 118 iface vmbr0 inet static
 119         address 192.168.0.3
 120         netmask  255.255.255.0
 121         bridge_ports eno1
 122         bridge_stp off
 123         bridge_fd 0
 124
 125 auto vxlan2
 126 iface vxlan2 inet manual
 127         vxlan-svcnodeip 225.20.1.1
 128         vxlan-physdev eno1
 129
 130 auto vmbr2
 131 iface vmbr2 inet manual
 132         bridge_ports vxlan2
 133         bridge_stp off
 134         bridge_fd 0
 135
 136
 137 auto vxlan3
 138 iface vxlan3 inet manual
 139         vxlan-svcnodeip 225.20.1.1
 140         vxlan-physdev eno1
 141
 142 auto vmbr3
 143 iface vmbr3 inet manual
 144         bridge_ports vxlan3
 145         bridge_stp off
 146         bridge_fd 0
 147 ----
 148
 149
 150 unicast mode
 151 ^^^^^^^^^^^^
 152
 153 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 154 The VXLAN is defined without a remote multicast group.
 155 Instead, all the remote VTEPs are associated with the all-zero address:
 156 a BUM frame will be duplicated to all these destinations.
 157 The VXLAN device will still learn remote addresses automatically using source-address learning.
 158
 159 * node1
 160
 161 ----
 162 auto eno1
 163 iface eno1 inet manual
 164
 165 auto vmbr0
 166 iface vmbr0 inet static
 167         address 192.168.0.1
 168         netmask  255.255.255.0
 169         bridge_ports eno1
 170         bridge_stp off
 171         bridge_fd 0
 172
 173
 174 auto vxlan2
 175 iface vxlan2 inet manual
 176         vxlan_remoteip 192.168.0.2
 177         vxlan_remoteip 192.168.0.3
 178
 179
 180 auto vmbr2
 181 iface vmbr2 inet manual
 182         bridge_ports vxlan2
 183         bridge_stp off
 184         bridge_fd 0
 185
 186
 187 auto vxlan3
 188 iface vxlan2 inet manual
 189         vxlan_remoteip 192.168.0.2
 190         vxlan_remoteip 192.168.0.3
 191
 192
 193 auto vmbr3
 194 iface vmbr3 inet manual
 195         bridge_ports vxlan3
 196         bridge_stp off
 197         bridge_fd 0
 198 ----
 199
 200
 201 * node2
 202
 203 ----
 204 auto eno1
 205 iface eno1 inet manual
 206
 207 auto vmbr0
 208 iface vmbr0 inet static
 209         address 192.168.0.2
 210         netmask  255.255.255.0
 211         bridge_ports eno1
 212         bridge_stp off
 213         bridge_fd 0
 214
 215 auto vxlan2
 216 iface vxlan2 inet manual
 217         vxlan_remoteip 192.168.0.1
 218         vxlan_remoteip 192.168.0.3
 219
 220
 221
 222 auto vmbr2
 223 iface vmbr2 inet manual
 224         bridge_ports vxlan2
 225         bridge_stp off
 226         bridge_fd 0
 227
 228 auto vxlan3
 229 iface vxlan2 inet manual
 230         vxlan_remoteip 192.168.0.1
 231         vxlan_remoteip 192.168.0.3
 232
 233
 234 auto vmbr3
 235 iface vmbr3 inet manual
 236         bridge_ports vxlan3
 237         bridge_stp off
 238         bridge_fd 0
 239 ----
 240
 241
 242 * node3
 243
 244 ----
 245 auto eno1
 246 iface eno1 inet manual
 247
 248 auto vmbr0
 249 iface vmbr0 inet static
 250         address 192.168.0.3
 251         netmask  255.255.255.0
 252         bridge_ports eno1
 253         bridge_stp off
 254         bridge_fd 0
 255
 256 auto vxlan2
 257 iface vxlan2 inet manual
 258         vxlan_remoteip 192.168.0.2
 259         vxlan_remoteip 192.168.0.3
 260
 261
 262
 263 auto vmbr2
 264 iface vmbr2 inet manual
 265         bridge_ports vxlan2
 266         bridge_stp off
 267         bridge_fd 0
 268
 269 auto vxlan3
 270 iface vxlan2 inet manual
 271         vxlan_remoteip 192.168.0.2
 272         vxlan_remoteip 192.168.0.3
 273
 274
 275 auto vmbr3
 276 iface vmbr3 inet manual
 277         bridge_ports vxlan3
 278         bridge_stp off
 279         bridge_fd 0
 280 ----
 281
 282
 283 bgp-evpn
 284 ^^^^^^^^
 285
 286 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 287 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 288
 289 The control plane used here is FRR, a bgp routing software.
 290 Each node in the proxmox cluster peer with each others nodes.
 291 For bigger networks, or multiple proxmox clusters,
 292 it's possible to use external bgp route reflector servers.
 293
 294 * node1
 295
 296 ----
 297 auto eno1
 298 iface eno1 inet manual
 299
 300 auto vmbr0
 301 iface vmbr0 inet static
 302         address 192.168.0.1
 303         netmask 255.255.255.0
 304         bridge_ports eno1
 305         bridge_stp off
 306         bridge_fd 0
 307
 308 auto vxlan2
 309 iface vxlan2 inet manual
 310         vxlan-local-tunnelip 192.168.0.1
 311         bridge-learning off
 312         bridge-arp-nd-suppress on
 313         bridge-unicast-flood off
 314         bridge-multicast-flood off
 315
 316
 317 auto vmbr2
 318 iface vmbr2 inet manual
 319         bridge_ports vxlan2
 320         bridge_stp off
 321         bridge_fd 0
 322
 323
 324 auto vxlan3
 325 iface vxlan3 inet manual
 326         vxlan-local-tunnelip 192.168.0.1
 327         bridge-learning off
 328         bridge-arp-nd-suppress on
 329         bridge-unicast-flood off
 330         bridge-multicast-flood off
 331
 332
 333 auto vmbr3
 334 iface vmbr3 inet manual
 335         bridge_ports vxlan3
 336         bridge_stp off
 337         bridge_fd 0
 338 ----
 339
 340
 341 /etc/frr/frr.conf
 342
 343 ----
 344 router bgp 1234
 345  no bgp default ipv4-unicast
 346  coalesce-time 1000
 347  neighbor 192.168.0.2 remote-as 1234
 348  neighbor 192.168.0.3 remote-as 1234
 349  !
 350  address-family l2vpn evpn
 351   neighbor 192.168.0.2 activate
 352   neighbor 192.168.0.3 activate
 353   advertise-all-vni
 354  exit-address-family
 355 !
 356 line vty
 357 !
 358 ----
 359
 360
 361 * node2
 362
 363 ----
 364 auto eno1
 365 iface eno1 inet manual
 366
 367 auto vmbr0
 368 iface vmbr0 inet static
 369         address 192.168.0.2
 370         netmask 255.255.255.0
 371         bridge_ports eno1
 372         bridge_stp off
 373         bridge_fd 0
 374
 375 auto vxlan2
 376 iface vxlan2 inet manual
 377         vxlan-local-tunnelip 192.168.0.2
 378         bridge-learning off
 379         bridge-arp-nd-suppress on
 380         bridge-unicast-flood off
 381         bridge-multicast-flood off
 382
 383
 384 auto vmbr2
 385 iface vmbr2 inet manual
 386         bridge_ports vxlan2
 387         bridge_stp off
 388         bridge_fd 0
 389
 390 auto vxlan3
 391 iface vxlan3 inet manual
 392         vxlan-local-tunnelip 192.168.0.2
 393         bridge-learning off
 394         bridge-arp-nd-suppress on
 395         bridge-unicast-flood off
 396         bridge-multicast-flood off
 397
 398
 399 auto vmbr3
 400 iface vmbr3 inet manual
 401         bridge_ports vxlan3
 402         bridge_stp off
 403         bridge_fd 0
 404 ----
 405
 406
 407 /etc/frr/frr.conf
 408
 409 ----
 410 router bgp 1234
 411  no bgp default ipv4-unicast
 412  coalesce-time 1000
 413  neighbor 192.168.0.1 remote-as 1234
 414  neighbor 192.168.0.3 remote-as 1234
 415  !
 416  address-family l2vpn evpn
 417   neighbor 192.168.0.1 activate
 418   neighbor 192.168.0.3 activate
 419   advertise-all-vni
 420  exit-address-family
 421 !
 422 line vty
 423 !
 424 ----
 425
 426
 427 * node3
 428
 429 ----
 430 auto eno1
 431 iface eno1 inet manual
 432
 433 auto vmbr0
 434 iface vmbr0 inet static
 435         address 192.168.0.2
 436         netmask 255.255.255.0
 437         bridge_ports eno1
 438         bridge_stp off
 439         bridge_fd 0
 440
 441 auto vxlan2
 442 iface vxlan2 inet manual
 443         vxlan-local-tunnelip 192.168.0.3
 444         bridge-learning off
 445         bridge-arp-nd-suppress on
 446         bridge-unicast-flood off
 447         bridge-multicast-flood off
 448
 449
 450 auto vmbr2
 451 iface vmbr2 inet manual
 452         bridge_ports vxlan2
 453         bridge_stp off
 454         bridge_fd 0
 455
 456 auto vxlan3
 457 iface vxlan3 inet manual
 458         vxlan-local-tunnelip 192.168.0.3
 459         bridge-learning off
 460         bridge-arp-nd-suppress on
 461         bridge-unicast-flood off
 462         bridge-multicast-flood off
 463
 464
 465 auto vmbr3
 466 iface vmbr3 inet manual
 467         bridge_ports vxlan3
 468         bridge_stp off
 469         bridge_fd 0
 470 ----
 471
 472
 473 /etc/frr/frr.conf
 474
 475
 476 ----
 477 router bgp 1234
 478  no bgp default ipv4-unicast
 479  coalesce-time 1000
 480  neighbor 192.168.0.1 remote-as 1234
 481  neighbor 192.168.0.2 remote-as 1234
 482  !
 483  address-family l2vpn evpn
 484   neighbor 192.168.0.1 activate
 485   neighbor 192.168.0.2 activate
 486   advertise-all-vni
 487  exit-address-family
 488 !
 489 line vty
 490 !
 491 ----
 492
 493 VXLAN layer3 routing with anycast gateway
 494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 495
 496 With this need, each vmbr bridge will be the gateway for the vm.
 497 Same vmbr on different node, will have same ip address and same mac address,
 498 to have working vm live migration and no network disruption.
 499
 500 VXLAN layer3 routing only work with FRR and non-aware bridge.
 501 (vlan aware bridge support is buggy currently).
 502
 503 asymmetric model
 504 ^^^^^^^^^^^^^^^^
 505
 506 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 507
 508 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 509 but only bridging on the egress.
 510 This results in bi-directional VXLAN traffic traveling on different VNIs
 511 in each direction (always the destination VNI) across the routed infrastructure.
 512
 513 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 514
 515
 516 sysctl.conf tuning
 517
 518 ----
 519 #enable routing
 520 net.ipv4.ip_forward=1
 521 net.ipv6.conf.all.forwarding=1
 522 ----
 523
 524 * node1
 525
 526 ----
 527 auto eno1
 528 iface eno1 inet manual
 529
 530 auto vmbr0
 531 iface vmbr0 inet static
 532          address 192.168.0.1
 533          netmask  255.255.255.0
 534          bridge_ports eno1
 535          bridge_stp off
 536          bridge_fd 0
 537
 538 auto vxlan2
 539 iface vxlan2 inet manual
 540         vxlan-local-tunnelip 192.168.0.1
 541         bridge-learning off
 542         bridge-arp-nd-suppress on
 543         bridge-unicast-flood off
 544         bridge-multicast-flood off
 545
 546
 547 auto vmbr2
 548 iface vmbr2 inet static
 549         address 10.0.2.254
 550         netmask 255.255.255.0
 551         hwaddress 44:39:39:FF:40:94
 552         bridge_ports vxlan2
 553         bridge_stp off
 554         bridge_fd 0
 555
 556
 557 auto vxlan3
 558 iface vxlan3 inet manual
 559         vxlan-local-tunnelip 192.168.0.1
 560         bridge-learning off
 561         bridge-arp-nd-suppress on
 562         bridge-unicast-flood off
 563         bridge-multicast-flood off
 564
 565
 566 auto vmbr3
 567 iface vmbr3 inet static
 568         address 10.0.3.254
 569         netmask 255.255.255.0
 570         hwaddress 44:39:39:FF:40:94
 571         bridge_ports vxlan3
 572         bridge_stp off
 573         bridge_fd 0
 574 ----
 575
 576
 577 frr.conf
 578
 579 ----
 580 router bgp 1234
 581  bgp router-id 192.168.0.1
 582  no bgp default ipv4-unicast
 583  coalesce-time 1000
 584  neighbor 192.168.0.2 remote-as 1234
 585  neighbor 192.168.0.3 remote-as 1234
 586  !
 587  address-family l2vpn evpn
 588   neighbor 192.168.0.2 activate
 589   neighbor 192.168.0.3 activate
 590   advertise-all-vni
 591  exit-address-family
 592 !
 593 line vty
 594 !
 595 ----
 596
 597
 598 * node2
 599
 600 ----
 601 auto eno1
 602 iface eno1 inet manual
 603
 604 auto vmbr0
 605 iface vmbr0 inet static
 606          address 192.168.0.2
 607          netmask  255.255.255.0
 608          bridge_ports eno1
 609          bridge_stp off
 610          bridge_fd 0
 611
 612 auto vxlan2
 613 iface vxlan2 inet manual
 614         vxlan-local-tunnelip 192.168.0.2
 615         bridge-learning off
 616         bridge-arp-nd-suppress on
 617         bridge-unicast-flood off
 618         bridge-multicast-flood off
 619
 620
 621 auto vmbr2
 622 iface vmbr2 inet static
 623         address 10.0.2.254
 624         netmask 255.255.255.0
 625         hwaddress 44:39:39:FF:40:94
 626         bridge_ports vxlan2
 627         bridge_stp off
 628         bridge_fd 0
 629
 630
 631 auto vxlan3
 632 iface vxlan3 inet manual
 633         vxlan-local-tunnelip 192.168.0.2
 634         bridge-learning off
 635         bridge-arp-nd-suppress on
 636         bridge-unicast-flood off
 637         bridge-multicast-flood off
 638
 639
 640 auto vmbr3
 641 iface vmbr3 inet static
 642         address 10.0.3.254
 643         netmask 255.255.255.0
 644         hwaddress 44:39:39:FF:40:94
 645         bridge_ports vxlan3
 646         bridge_stp off
 647         bridge_fd 0
 648 ----
 649
 650
 651 frr.conf
 652
 653 ----
 654 router bgp 1234
 655  bgp router-id 192.168.0.2
 656  no bgp default ipv4-unicast
 657  coalesce-time 1000
 658  neighbor 192.168.0.1 remote-as 1234
 659  neighbor 192.168.0.3 remote-as 1234
 660  !
 661  address-family l2vpn evpn
 662   neighbor 192.168.0.1 activate
 663   neighbor 192.168.0.3 activate
 664   advertise-all-vni
 665  exit-address-family
 666 !
 667 line vty
 668 !
 669 ----
 670
 671
 672 * node3
 673
 674 ----
 675 auto eno1
 676 iface eno1 inet manual
 677
 678 auto vmbr0
 679 iface vmbr0 inet static
 680          address 192.168.0.3
 681          netmask  255.255.255.0
 682          bridge_ports eno1
 683          bridge_stp off
 684          bridge_fd 0
 685
 686 auto vxlan2
 687 iface vxlan2 inet manual
 688         vxlan-local-tunnelip 192.168.0.3
 689         bridge-learning off
 690         bridge-arp-nd-suppress on
 691         bridge-unicast-flood off
 692         bridge-multicast-flood off
 693
 694
 695 auto vmbr2
 696 iface vmbr2 inet static
 697         address 10.0.2.254
 698         netmask 255.255.255.0
 699         hwaddress 44:39:39:FF:40:94
 700         bridge_ports vxlan2
 701         bridge_stp off
 702         bridge_fd 0
 703
 704
 705 auto vxlan3
 706 iface vxlan3 inet manual
 707         vxlan-local-tunnelip 192.168.0.3
 708         bridge-learning off
 709         bridge-arp-nd-suppress on
 710         bridge-unicast-flood off
 711         bridge-multicast-flood off
 712
 713
 714 auto vmbr3
 715 iface vmbr3 inet static
 716         address 10.0.3.254
 717         netmask 255.255.255.0
 718         hwaddress 44:39:39:FF:40:94
 719         bridge_ports vxlan3
 720         bridge_stp off
 721         bridge_fd 0
 722 ----
 723
 724
 725 frr.conf
 726
 727 ----
 728 router bgp 1234
 729  bgp router-id 192.168.0.3
 730  no bgp default ipv4-unicast
 731  coalesce-time 1000
 732  neighbor 192.168.0.1 remote-as 1234
 733  neighbor 192.168.0.2 remote-as 1234
 734  !
 735  address-family l2vpn evpn
 736   neighbor 192.168.0.1 activate
 737   neighbor 192.168.0.2 activate
 738   advertise-all-vni
 739  exit-address-family
 740 !
 741 line vty
 742 !
 743 ----
 744
 745
 746 symmetric model
 747 ^^^^^^^^^^^^^^^
 748
 749 With this model, you don't need to have all vxlan on all nodes.
 750 This model will also be needed to route traffic to an external router.
 751
 752 The symmetric model routes and bridges on both the ingress and the egress leafs.
 753 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 754 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 755 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 756 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 757
 758 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 759
 760 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 761
 762 sysctl.conf tuning
 763
 764 ----
 765 #enable routing
 766 net.ipv4.ip_forward=1
 767 net.ipv6.conf.all.forwarding=1
 768 #disable reverse path filtering
 769 net.ipv4.conf.default.rp_filter=0
 770 net.ipv4.conf.all.rp_filter=0
 771 #allow frr to work with vrf (kernel >4.14 bug)
 772 net.ipv4.tcp_l3mdev_accept=1
 773 ----
 774
 775 * node1
 776
 777 ----
 778 auto vrf1
 779 iface vrf1
 780     vrf-table auto
 781
 782 auto eno1
 783 iface eno1 inet manual
 784
 785 auto vmbr0
 786 iface vmbr0 inet static
 787          address 192.168.0.1
 788          netmask  255.255.255.0
 789          bridge_ports eno1
 790          bridge_stp off
 791          bridge_fd 0
 792
 793 auto vxlan2
 794 iface vxlan2 inet manual
 795         vxlan-local-tunnelip 192.168.0.1
 796         bridge-learning off
 797         bridge-arp-nd-suppress on
 798         bridge-unicast-flood off
 799         bridge-multicast-flood off
 800
 801 auto vmbr2
 802 iface vmbr2 inet static
 803         bridge_ports vxlan2
 804         bridge_stp off
 805         bridge_fd 0
 806         address 10.0.2.254
 807         netmask 255.255.255.0
 808         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 809         vrf vrf1
 810
 811 auto vxlan3
 812 iface vxlan3 inet manual
 813         vxlan-local-tunnelip 192.168.0.1
 814         bridge-learning off
 815         bridge-arp-nd-suppress on
 816         bridge-unicast-flood off
 817         bridge-multicast-flood off
 818
 819 auto vmbr3
 820 iface vmbr3 inet static
 821         bridge_ports vxlan3
 822         bridge_stp off
 823         bridge_fd 0
 824         address 10.0.3.254
 825         netmask 255.255.255.0
 826         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 827         vrf vrf1
 828
 829 #interconnect vxlan-vfr l3vni
 830 auto vxlan4000
 831 iface vxlan4000 inet manual
 832         vxlan-local-tunnelip 192.168.0.1
 833         bridge-learning off
 834         bridge-arp-nd-suppress on
 835         bridge-unicast-flood off
 836         bridge-multicast-flood off
 837
 838
 839 auto vmbr4000
 840 iface vmbr4000 inet manual
 841         bridge_ports vxlan4000
 842         bridge_stp off
 843         bridge_fd 0
 844         hwaddress 44:39:39:FF:40:90  #must be different on each node
 845         vrf vrf1
 846 ----
 847
 848 frr.conf
 849
 850 ----
 851 vrf vrf1
 852  vni 4000
 853 !
 854 router bgp 1234
 855  bgp router-id 192.168.0.1
 856  no bgp default ipv4-unicast
 857  coalesce-time 1000
 858  neighbor 192.168.0.2 remote-as 1234
 859  neighbor 192.168.0.3 remote-as 1234
 860  !
 861  address-family l2vpn evpn
 862   neighbor 192.168.0.2 activate
 863   neighbor 192.168.0.3 activate
 864   advertise-all-vni
 865  exit-address-family
 866 !
 867 router bgp 1234 vrf vrf1
 868 !
 869  bgp router-id 192.168.0.1
 870  !
 871  address-family ipv4 unicast
 872   redistribute connected
 873  exit-address-family
 874  !
 875  address-family l2vpn evpn
 876   advertise ipv4 unicast
 877  exit-address-family
 878 !
 879 line vty
 880 !
 881 ----
 882
 883
 884 * node2
 885
 886 ----
 887 auto vrf1
 888 iface vrf1
 889     vrf-table auto
 890
 891 auto eno1
 892 iface eno1 inet manual
 893
 894 auto vmbr0
 895 iface vmbr0 inet static
 896          address 192.168.0.2
 897          netmask  255.255.255.0
 898          bridge_ports eno1
 899          bridge_stp off
 900          bridge_fd 0
 901
 902 auto vxlan2
 903 iface vxlan2 inet manual
 904         vxlan-local-tunnelip 192.168.0.2
 905         bridge-learning off
 906         bridge-arp-nd-suppress on
 907         bridge-unicast-flood off
 908         bridge-multicast-flood off
 909
 910 auto vmbr2
 911 iface vmbr2 inet static
 912         bridge_ports vxlan2
 913         bridge_stp off
 914         bridge_fd 0
 915         address 10.0.2.254
 916         netmask 255.255.255.0
 917         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 918         vrf vrf1
 919
 920 auto vxlan3
 921 iface vxlan3 inet manual
 922         vxlan-local-tunnelip 192.168.0.2
 923         bridge-learning off
 924         bridge-arp-nd-suppress on
 925         bridge-unicast-flood off
 926         bridge-multicast-flood off
 927
 928 auto vmbr3
 929 iface vmbr3 inet static
 930         bridge_ports vxlan3
 931         bridge_stp off
 932         bridge_fd 0
 933         address 10.0.3.254
 934         netmask 255.255.255.0
 935         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 936         vrf vrf1
 937
 938 #interconnect vxlan-vfr l3vni
 939 auto vxlan4000
 940 iface vxlan4000 inet manual
 941         vxlan-local-tunnelip 192.168.0.2
 942         bridge-learning off
 943         bridge-arp-nd-suppress on
 944         bridge-unicast-flood off
 945         bridge-multicast-flood off
 946
 947
 948 auto vmbr4000
 949 iface vmbr4000 inet manual
 950         bridge_ports vxlan4000
 951         bridge_stp off
 952         bridge_fd 0
 953         hwaddress 44:39:39:FF:40:91  #must be different on each node
 954         vrf vrf1
 955 ----
 956
 957
 958 frr.conf
 959
 960 ----
 961 vrf vrf1
 962  vni 4000
 963 !
 964 router bgp 1234
 965  bgp router-id 192.168.0.2
 966  no bgp default ipv4-unicast
 967  coalesce-time 1000
 968  neighbor 192.168.0.1 remote-as 1234
 969  neighbor 192.168.0.3 remote-as 1234
 970  !
 971  address-family l2vpn evpn
 972   neighbor 192.168.0.1 activate
 973   neighbor 192.168.0.3 activate
 974   advertise-all-vni
 975  exit-address-family
 976 !
 977 router bgp 1234 vrf vrf1
 978 !
 979  bgp router-id 192.168.0.2
 980  !
 981  address-family ipv4 unicast
 982   redistribute connected
 983  exit-address-family
 984  !
 985  address-family l2vpn evpn
 986   advertise ipv4 unicast
 987  exit-address-family
 988 !
 989 line vty
 990 !
 991 ----
 992
 993
 994 * node3
 995
 996 ----
 997 auto vrf1
 998 iface vrf1
 999     vrf-table auto
1000
1001 auto eno1
1002 iface eno1 inet manual
1003
1004 auto vmbr0
1005 iface vmbr0 inet static
1006          address 192.168.0.3
1007          netmask  255.255.255.0
1008          bridge_ports eno1
1009          bridge_stp off
1010          bridge_fd 0
1011
1012 auto vxlan2
1013 iface vxlan2 inet manual
1014         vxlan-local-tunnelip 192.168.0.3
1015         bridge-learning off
1016         bridge-arp-nd-suppress on
1017         bridge-unicast-flood off
1018         bridge-multicast-flood off
1019
1020 auto vmbr2
1021 iface vmbr2 inet static
1022         bridge_ports vxlan2
1023         bridge_stp off
1024         bridge_fd 0
1025         address 10.0.2.254
1026         netmask 255.255.255.0
1027         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1028         vrf vrf1
1029
1030 auto vxlan3
1031 iface vxlan3 inet manual
1032         vxlan-local-tunnelip 192.168.0.3
1033         bridge-learning off
1034         bridge-arp-nd-suppress on
1035         bridge-unicast-flood off
1036         bridge-multicast-flood off
1037
1038 auto vmbr3
1039 iface vmbr3 inet static
1040         bridge_ports vxlan3
1041         bridge_stp off
1042         bridge_fd 0
1043         address 10.0.3.254
1044         netmask 255.255.255.0
1045         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1046         vrf vrf1
1047
1048 #interconnect vxlan-vfr l3vni
1049 auto vxlan4000
1050 iface vxlan4000 inet manual
1051         vxlan-local-tunnelip 192.168.0.3
1052         bridge-learning off
1053         bridge-arp-nd-suppress on
1054         bridge-unicast-flood off
1055         bridge-multicast-flood off
1056
1057
1058 auto vmbr4000
1059 iface vmbr4000 inet manual
1060         bridge_ports vxlan4000
1061         bridge_stp off
1062         bridge_fd 0
1063         hwaddress 44:39:39:FF:40:92  #must be different on each node
1064         vrf vrf1
1065 ----
1066
1067
1068 frr.conf
1069
1070 ----
1071 vrf vrf1
1072  vni 4000
1073 !
1074 router bgp 1234
1075  bgp router-id 192.168.0.3
1076  no bgp default ipv4-unicast
1077  coalesce-time 1000
1078  neighbor 192.168.0.1 remote-as 1234
1079  neighbor 192.168.0.2 remote-as 1234
1080  !
1081  address-family l2vpn evpn
1082   neighbor 192.168.0.1 activate
1083   neighbor 192.168.0.2 activate
1084   advertise-all-vni
1085  exit-address-family
1086 !
1087 router bgp 1234 vrf vrf1
1088 !
1089  bgp router-id 192.168.0.3
1090  !
1091  address-family ipv4 unicast
1092   redistribute connected
1093  exit-address-family
1094  !
1095  address-family l2vpn evpn
1096   advertise ipv4 unicast
1097  exit-address-family
1098 !
1099 line vty
1100 !
1101 ----
1102
1103 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1104 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1105 Routing to outside need the symmetric model.
1106
1107 1 gateway node
1108 ^^^^^^^^^^^^^^
1109 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1110 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1111 and announce this default gw to other proxmox nodes.
1112
1113
1114 *node1
1115
1116 ----
1117 auto vrf1
1118 iface vrf1
1119     vrf-table auto
1120
1121 auto eno1
1122 iface eno1 inet manual
1123
1124 auto vmbr0
1125 iface vmbr0 inet static
1126          address 192.168.0.1
1127          netmask  255.255.255.0
1128          bridge_ports eno1
1129          bridge_stp off
1130          bridge_fd 0
1131
1132 auto eno2
1133 iface eno2
1134         address 172.16.0.1
1135         netmask 255.255.255.0
1136         vrf vrf1
1137         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1138         #if you have multiple external routers, you can use ecmp balancing
1139         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1140
1141 auto vxlan2
1142 iface vxlan2 inet manual
1143         vxlan-local-tunnelip 192.168.0.1
1144         bridge-learning off
1145         bridge-arp-nd-suppress on
1146         bridge-unicast-flood off
1147         bridge-multicast-flood off
1148
1149 auto vmbr2
1150 iface vmbr2 inet static
1151         bridge_ports vxlan2
1152         bridge_stp off
1153         bridge_fd 0
1154         address 10.0.2.254
1155         netmask 255.255.255.0
1156         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1157         vrf vrf1
1158
1159 auto vxlan3
1160 iface vxlan3 inet manual
1161         vxlan-local-tunnelip 192.168.0.1
1162         bridge-learning off
1163         bridge-arp-nd-suppress on
1164         bridge-unicast-flood off
1165         bridge-multicast-flood off
1166
1167 auto vmbr3
1168 iface vmbr3 inet static
1169         bridge_ports vxlan3
1170         bridge_stp off
1171         bridge_fd 0
1172         address 10.0.3.254
1173         netmask 255.255.255.0
1174         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1175         vrf vrf1
1176
1177 #interconnect vxlan-vfr l3vni
1178 auto vxlan4000
1179 iface vxlan4000 inet manual
1180         vxlan-local-tunnelip 192.168.0.1
1181         bridge-learning off
1182         bridge-arp-nd-suppress on
1183         bridge-unicast-flood off
1184         bridge-multicast-flood off
1185
1186 auto vmbr4000
1187 iface vmbr4000 inet manual
1188         bridge_ports vxlan4000
1189         bridge_stp off
1190         bridge_fd 0
1191         hwaddress 44:39:39:FF:40:90  #must be different on each node
1192         vrf vrf1
1193 ----
1194
1195
1196 frr.conf
1197
1198 ----
1199 vrf vrf1
1200  vni 4000
1201 !
1202 router bgp 1234
1203  bgp router-id 192.168.0.1
1204  no bgp default ipv4-unicast
1205  coalesce-time 1000
1206  neighbor 192.168.0.2 remote-as 1234
1207  neighbor 192.168.0.3 remote-as 1234
1208  !
1209  address-family l2vpn evpn
1210   neighbor 192.168.0.2 activate
1211   neighbor 192.168.0.3 activate
1212   advertise-all-vni
1213  exit-address-family
1214 !
1215 router bgp 1234 vrf vrf1
1216 !
1217  bgp router-id 172.16.0.1
1218  !
1219  address-family ipv4 unicast
1220   redistribute connected
1221   redistribute kernel !announce your default gw to all nodes
1222  exit-address-family
1223  !
1224  address-family l2vpn evpn
1225   advertise ipv4 unicast
1226  exit-address-family
1227 !
1228 line vty
1229 !
1230 ----
1231
1232
1233 * node2
1234
1235 ----
1236 auto vrf1
1237 iface vrf1
1238     vrf-table auto
1239
1240 auto eno1
1241 iface eno1 inet manual
1242
1243 auto vmbr0
1244 iface vmbr0 inet static
1245          address 192.168.0.2
1246          netmask  255.255.255.0
1247          bridge_ports eno1
1248          bridge_stp off
1249          bridge_fd 0
1250
1251 auto vxlan2
1252 iface vxlan2 inet manual
1253         vxlan-local-tunnelip 192.168.0.2
1254         bridge-learning off
1255         bridge-arp-nd-suppress on
1256         bridge-unicast-flood off
1257         bridge-multicast-flood off
1258
1259 auto vmbr2
1260 iface vmbr2 inet static
1261         bridge_ports vxlan2
1262         bridge_stp off
1263         bridge_fd 0
1264         address 10.0.2.254
1265         netmask 255.255.255.0
1266         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1267         vrf vrf1
1268
1269 auto vxlan3
1270 iface vxlan3 inet manual
1271         vxlan-local-tunnelip 192.168.0.2
1272         bridge-learning off
1273         bridge-arp-nd-suppress on
1274         bridge-unicast-flood off
1275         bridge-multicast-flood off
1276
1277 auto vmbr3
1278 iface vmbr3 inet static
1279         bridge_ports vxlan3
1280         bridge_stp off
1281         bridge_fd 0
1282         address 10.0.3.254
1283         netmask 255.255.255.0
1284         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1285         vrf vrf1
1286
1287 #interconnect vxlan-vfr l3vni
1288 auto vxlan4000
1289 iface vxlan4000 inet manual
1290         vxlan-local-tunnelip 192.168.0.2
1291         bridge-learning off
1292         bridge-arp-nd-suppress on
1293         bridge-unicast-flood off
1294         bridge-multicast-flood off
1295
1296
1297 auto vmbr4000
1298 iface vmbr4000 inet manual
1299         bridge_ports vxlan4000
1300         bridge_stp off
1301         bridge_fd 0
1302         hwaddress 44:39:39:FF:40:91  #must be different on each node
1303         vrf vrf1
1304 ----
1305
1306
1307 frr.conf
1308
1309 ----
1310 vrf vrf1
1311  vni 4000
1312 !
1313 router bgp 1234
1314  bgp router-id 192.168.0.2
1315  no bgp default ipv4-unicast
1316  coalesce-time 1000
1317  neighbor 192.168.0.1 remote-as 1234
1318  neighbor 192.168.0.3 remote-as 1234
1319  !
1320  address-family l2vpn evpn
1321   neighbor 192.168.0.1 activate
1322   neighbor 192.168.0.3 activate
1323   advertise-all-vni
1324  exit-address-family
1325 !
1326 router bgp 1234 vrf vrf1
1327 !
1328  bgp router-id 192.168.0.2
1329  !
1330  address-family ipv4 unicast
1331   redistribute connected
1332  exit-address-family
1333  !
1334  address-family l2vpn evpn
1335   advertise ipv4 unicast
1336  exit-address-family
1337 !
1338 line vty
1339 !
1340 ----
1341
1342
1343 * node3
1344
1345 ----
1346 auto vrf1
1347 iface vrf1
1348     vrf-table auto
1349
1350 auto eno1
1351 iface eno1 inet manual
1352
1353 auto vmbr0
1354 iface vmbr0 inet static
1355          address 192.168.0.3
1356          netmask  255.255.255.0
1357          bridge_ports eno1
1358          bridge_stp off
1359          bridge_fd 0
1360
1361 auto vxlan2
1362 iface vxlan2 inet manual
1363         vxlan-local-tunnelip 192.168.0.3
1364         bridge-learning off
1365         bridge-arp-nd-suppress on
1366         bridge-unicast-flood off
1367         bridge-multicast-flood off
1368
1369 auto vmbr2
1370 iface vmbr2 inet static
1371         bridge_ports vxlan2
1372         bridge_stp off
1373         bridge_fd 0
1374         address 10.0.2.254
1375         netmask 255.255.255.0
1376         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1377         vrf vrf1
1378
1379 auto vxlan3
1380 iface vxlan3 inet manual
1381         vxlan-local-tunnelip 192.168.0.3
1382         bridge-learning off
1383         bridge-arp-nd-suppress on
1384         bridge-unicast-flood off
1385         bridge-multicast-flood off
1386
1387 auto vmbr3
1388 iface vmbr3 inet static
1389         bridge_ports vxlan3
1390         bridge_stp off
1391         bridge_fd 0
1392         address 10.0.3.254
1393         netmask 255.255.255.0
1394         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1395         vrf vrf1
1396
1397 #interconnect vxlan-vfr l3vni
1398 auto vxlan4000
1399 iface vxlan4000 inet manual
1400         vxlan-local-tunnelip 192.168.0.3
1401         bridge-learning off
1402         bridge-arp-nd-suppress on
1403         bridge-unicast-flood off
1404         bridge-multicast-flood off
1405
1406
1407 auto vmbr4000
1408 iface vmbr4000 inet manual
1409         bridge_ports vxlan4000
1410         bridge_stp off
1411         bridge_fd 0
1412         hwaddress 44:39:39:FF:40:92  #must be different on each node
1413         vrf vrf1
1414 ----
1415
1416
1417 frr.conf
1418
1419 ----
1420 vrf vrf1
1421  vni 4000
1422 !
1423 router bgp 1234
1424  bgp router-id 192.168.0.3
1425  no bgp default ipv4-unicast
1426  coalesce-time 1000
1427  neighbor 192.168.0.1 remote-as 1234
1428  neighbor 192.168.0.2 remote-as 1234
1429  !
1430  address-family l2vpn evpn
1431   neighbor 192.168.0.1 activate
1432   neighbor 192.168.0.2 activate
1433   advertise-all-vni
1434  exit-address-family
1435 !
1436 router bgp 1234 vrf vrf1
1437 !
1438  bgp router-id 192.168.0.3
1439  !
1440  address-family ipv4 unicast
1441   redistribute connected
1442  exit-address-family
1443  !
1444  address-family l2vpn evpn
1445   advertise ipv4 unicast
1446  exit-address-family
1447 !
1448 line vty
1449 !
1450 ----
1451
1452 multiple gateway nodes
1453 ^^^^^^^^^^^^^^^^^^^^^^
1454 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1455 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1456 and announce this default gw.
1457 The external router have ecmp routes to all proxmox nodes.(balancing).
1458 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1459 vxlan the packet to final destination.
1460
1461 *node1
1462
1463 ----
1464 auto vrf1
1465 iface vrf1
1466     vrf-table auto
1467
1468 auto eno1
1469 iface eno1 inet manual
1470
1471 auto vmbr0
1472 iface vmbr0 inet static
1473          address 192.168.0.1
1474          netmask  255.255.255.0
1475          bridge_ports eno1
1476          bridge_stp off
1477          bridge_fd 0
1478
1479 auto eno2
1480 iface eno2
1481         address 172.16.0.1
1482         netmask 255.255.255.0
1483         vrf vrf1
1484         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1485         #if you have multiple external routers, you can use ecmp balancing
1486         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1487
1488 auto vxlan2
1489 iface vxlan2 inet manual
1490         vxlan-local-tunnelip 192.168.0.1
1491         bridge-learning off
1492         bridge-arp-nd-suppress on
1493         bridge-unicast-flood off
1494         bridge-multicast-flood off
1495
1496 auto vmbr2
1497 iface vmbr2 inet static
1498         bridge_ports vxlan2
1499         bridge_stp off
1500         bridge_fd 0
1501         address 10.0.2.254
1502         netmask 255.255.255.0
1503         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1504         vrf vrf1
1505
1506 auto vxlan3
1507 iface vxlan3 inet manual
1508         vxlan-local-tunnelip 192.168.0.1
1509         bridge-learning off
1510         bridge-arp-nd-suppress on
1511         bridge-unicast-flood off
1512         bridge-multicast-flood off
1513
1514 auto vmbr3
1515 iface vmbr3 inet static
1516         bridge_ports vxlan3
1517         bridge_stp off
1518         bridge_fd 0
1519         address 10.0.3.254
1520         netmask 255.255.255.0
1521         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1522         vrf vrf1
1523
1524 #interconnect vxlan-vfr l3vni
1525 auto vxlan4000
1526 iface vxlan4000 inet manual
1527         vxlan-local-tunnelip 192.168.0.1
1528         bridge-learning off
1529         bridge-arp-nd-suppress on
1530         bridge-unicast-flood off
1531         bridge-multicast-flood off
1532
1533 auto vmbr4000
1534 iface vmbr4000 inet manual
1535         bridge_ports vxlan4000
1536         bridge_stp off
1537         bridge_fd 0
1538         hwaddress 44:39:39:FF:40:90  #must be different on each node
1539         vrf vrf1
1540 ----
1541
1542
1543 frr.conf
1544
1545 ----
1546 vrf vrf1
1547  vni 4000
1548 !
1549 router bgp 1234
1550  bgp router-id 192.168.0.1
1551  no bgp default ipv4-unicast
1552  coalesce-time 1000
1553  neighbor 192.168.0.2 remote-as 1234
1554  neighbor 192.168.0.3 remote-as 1234
1555  !
1556  address-family l2vpn evpn
1557   neighbor 192.168.0.2 activate
1558   neighbor 192.168.0.3 activate
1559   advertise-all-vni
1560  exit-address-family
1561 !
1562 router bgp 1234 vrf vrf1
1563 !
1564  bgp router-id 172.16.0.1
1565  !
1566  address-family ipv4 unicast
1567   redistribute connected
1568   redistribute kernel !announce your default gw to all nodes
1569  exit-address-family
1570  !
1571  address-family l2vpn evpn
1572   advertise ipv4 unicast
1573  exit-address-family
1574 !
1575 line vty
1576 !
1577 ----
1578
1579
1580 * node2
1581
1582 ----
1583 auto vrf1
1584 iface vrf1
1585     vrf-table auto
1586
1587 auto eno1
1588 iface eno1 inet manual
1589
1590 auto vmbr0
1591 iface vmbr0 inet static
1592          address 192.168.0.2
1593          netmask  255.255.255.0
1594          bridge_ports eno1
1595          bridge_stp off
1596          bridge_fd 0
1597
1598 auto eno2
1599 iface eno2
1600         address 172.16.0.3
1601         netmask 255.255.255.0
1602         vrf vrf1
1603         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1604         #if you have multiple external routers, you can use ecmp balancing
1605         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1606
1607 auto vxlan2
1608 iface vxlan2 inet manual
1609         vxlan-local-tunnelip 192.168.0.2
1610         bridge-learning off
1611         bridge-arp-nd-suppress on
1612         bridge-unicast-flood off
1613         bridge-multicast-flood off
1614
1615 auto vmbr2
1616 iface vmbr2 inet static
1617         bridge_ports vxlan2
1618         bridge_stp off
1619         bridge_fd 0
1620         address 10.0.2.254
1621         netmask 255.255.255.0
1622         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1623         vrf vrf1
1624
1625 auto vxlan3
1626 iface vxlan3 inet manual
1627         vxlan-local-tunnelip 192.168.0.2
1628         bridge-learning off
1629         bridge-arp-nd-suppress on
1630         bridge-unicast-flood off
1631         bridge-multicast-flood off
1632
1633 auto vmbr3
1634 iface vmbr3 inet static
1635         bridge_ports vxlan3
1636         bridge_stp off
1637         bridge_fd 0
1638         address 10.0.3.254
1639         netmask 255.255.255.0
1640         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1641         vrf vrf1
1642
1643 #interconnect vxlan-vfr l3vni
1644 auto vxlan4000
1645 iface vxlan4000 inet manual
1646         vxlan-local-tunnelip 192.168.0.2
1647         bridge-learning off
1648         bridge-arp-nd-suppress on
1649         bridge-unicast-flood off
1650         bridge-multicast-flood off
1651
1652
1653 auto vmbr4000
1654 iface vmbr4000 inet manual
1655         bridge_ports vxlan4000
1656         bridge_stp off
1657         bridge_fd 0
1658         hwaddress 44:39:39:FF:40:91  #must be different on each node
1659         vrf vrf1
1660 ----
1661
1662
1663 frr.conf
1664
1665 ----
1666 vrf vrf1
1667  vni 4000
1668 !
1669 router bgp 1234
1670  bgp router-id 192.168.0.2
1671  no bgp default ipv4-unicast
1672  coalesce-time 1000
1673  neighbor 192.168.0.1 remote-as 1234
1674  neighbor 192.168.0.3 remote-as 1234
1675  !
1676  address-family l2vpn evpn
1677   neighbor 192.168.0.1 activate
1678   neighbor 192.168.0.3 activate
1679   advertise-all-vni
1680  exit-address-family
1681 !
1682 router bgp 1234 vrf vrf1
1683 !
1684  bgp router-id 172.16.0.2
1685  !
1686  address-family ipv4 unicast
1687   redistribute connected
1688   redistribute kernel !announce your default gw to all nodes
1689  exit-address-family
1690  !
1691  address-family l2vpn evpn
1692   advertise ipv4 unicast
1693  exit-address-family
1694 !
1695 line vty
1696 !
1697 ----
1698
1699
1700 * node3
1701
1702 ----
1703 auto vrf1
1704 iface vrf1
1705     vrf-table auto
1706
1707 auto eno1
1708 iface eno1 inet manual
1709
1710 auto vmbr0
1711 iface vmbr0 inet static
1712          address 192.168.0.3
1713          netmask  255.255.255.0
1714          bridge_ports eno1
1715          bridge_stp off
1716          bridge_fd 0
1717
1718 auto eno2
1719 iface eno2
1720         address 172.16.0.3
1721         netmask 255.255.255.0
1722         vrf vrf1
1723         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1724         #if you have multiple external routers, you can use ecmp balancing
1725         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1726
1727 auto vxlan2
1728 iface vxlan2 inet manual
1729         vxlan-local-tunnelip 192.168.0.3
1730         bridge-learning off
1731         bridge-arp-nd-suppress on
1732         bridge-unicast-flood off
1733         bridge-multicast-flood off
1734
1735 auto vmbr2
1736 iface vmbr2 inet static
1737         bridge_ports vxlan2
1738         bridge_stp off
1739         bridge_fd 0
1740         address 10.0.2.254
1741         netmask 255.255.255.0
1742         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1743         vrf vrf1
1744
1745 auto vxlan3
1746 iface vxlan3 inet manual
1747         vxlan-local-tunnelip 192.168.0.3
1748         bridge-learning off
1749         bridge-arp-nd-suppress on
1750         bridge-unicast-flood off
1751         bridge-multicast-flood off
1752
1753 auto vmbr3
1754 iface vmbr3 inet static
1755         bridge_ports vxlan3
1756         bridge_stp off
1757         bridge_fd 0
1758         address 10.0.3.254
1759         netmask 255.255.255.0
1760         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1761         vrf vrf1
1762
1763 #interconnect vxlan-vfr l3vni
1764 auto vxlan4000
1765 iface vxlan4000 inet manual
1766         vxlan-local-tunnelip 192.168.0.3
1767         bridge-learning off
1768         bridge-arp-nd-suppress on
1769         bridge-unicast-flood off
1770         bridge-multicast-flood off
1771
1772
1773 auto vmbr4000
1774 iface vmbr4000 inet manual
1775         bridge_ports vxlan4000
1776         bridge_stp off
1777         bridge_fd 0
1778         hwaddress 44:39:39:FF:40:92  #must be different on each node
1779         vrf vrf1
1780 ----
1781
1782
1783 frr.conf
1784
1785 ----
1786 vrf vrf1
1787  vni 4000
1788 !
1789 router bgp 1234
1790  bgp router-id 192.168.0.3
1791  no bgp default ipv4-unicast
1792  coalesce-time 1000
1793  neighbor 192.168.0.1 remote-as 1234
1794  neighbor 192.168.0.2 remote-as 1234
1795  !
1796  address-family l2vpn evpn
1797   neighbor 192.168.0.1 activate
1798   neighbor 192.168.0.2 activate
1799   advertise-all-vni
1800  exit-address-family
1801 !
1802 router bgp 1234 vrf vrf1
1803 !
1804  bgp router-id 172.16.0.3
1805  !
1806  address-family ipv4 unicast
1807   redistribute connected
1808   redistribute kernel !announce your default gw to all nodes
1809  exit-address-family
1810  !
1811  address-family l2vpn evpn
1812   advertise ipv4 unicast
1813  exit-address-family
1814 !
1815 line vty
1816 !
1817 ----
1818
1819 Note
1820 ^^^^
1821
1822 If your external router don't support ecmp to reach multiple proxmox nodes,
1823 you can setup an HA floating vip on proxmox nodes with vrrp
1824
1825 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1826 Node1 is the primary and failover to node2 in case of failure.
1827
1828
1829 * node1
1830
1831 ----
1832 auto eno2
1833 iface eno2
1834         address 172.16.0.1
1835         netmask 255.255.255.0
1836         vrf vrf1
1837         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1838         vrrp-id 1
1839         vrrp-priority 1
1840         vrrp-virtual-ip 172.16.0.10
1841 ----
1842
1843 * node2
1844
1845 ----
1846 auto eno2
1847 iface eno2
1848         address 172.16.0.2
1849         netmask 255.255.255.0
1850         vrf vrf1
1851         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1852         vrrp-id 1
1853         vrrp-priority 2
1854         vrrp-virtual-ip 172.16.0.10
1855 ----
1856
1857