vxlan-and-evpn.adoc

   1
   2 ////
   3
   4 This is currently not included, because
   5 - it requires ifupdown2
   6 - routing needs more documentation
   7
   8 ////
   9
  10
  11 VXLAN layer2 with vlan unware linux bridges
  12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  13
  14 VXLAN is an overlay network to carry Ethernet traffic over an existing IP network
  15 while accommodating a very large number of tenants. It is defined in RFC 7348.
  16 Each overlay network is known as a VXLAN Segment and identified by a unique
  17 24-bit segment ID called a VXLAN Network Identifier (VNI).
  18
  19 VXLAN encapsulation add 50bytes overhead, so you need to increase mtu on your host
  20 physical interfaces to 1550 at minimum. (or decrease mtu inside your vms to 1450)
  21
  22 For BUM traffic (broadcast / unknown unicast traffic, multicast),
  23 we have 3 differents vxlan setup modes : multicast, unicast, bgp-evpn
  24
  25 image::images/vxlan-l2-vlanunaware.svg["vxlan l2 bridge vlan unaware",align="center"]
  26
  27 multicast mode
  28 ^^^^^^^^^^^^^^
  29
  30 This scenario relies in head end replication, meaning that end host in case
  31 of not having any entry for the destination MAC address will send out an ARP
  32 to other devices / VTEPs in the VXLAN network.
  33 This is done by sending the request to the VXLAN multicast group,
  34 remote VTEPs will get the packet and answer accordingly direct to the originating VTEP.
  35
  36
  37 * node1
  38
  39 ----
  40 auto eno1
  41 iface eno1 inet manual
  42         mtu 1550
  43
  44 auto vmbr0
  45 iface vmbr0 inet static
  46         address 192.168.0.1
  47         netmask  255.255.255.0
  48         bridge_ports eno1
  49         bridge_stp off
  50         bridge_fd 0
  51
  52 auto vxlan2
  53 iface vxlan2 inet manual
  54         vxlan-svcnodeip 225.20.1.1
  55         vxlan-physdev eno1
  56
  57 auto vmbr2
  58 iface vmbr2 inet manual
  59         bridge_ports vxlan2
  60         bridge_stp off
  61         bridge_fd 0
  62
  63 auto vxlan3
  64 iface vxlan3 inet manual
  65         vxlan-svcnodeip 225.20.1.1
  66         vxlan-physdev eno1
  67
  68 auto vmbr3
  69 iface vmbr3 inet manual
  70         bridge_ports vxlan3
  71         bridge_stp off
  72         bridge_fd 0
  73 ----
  74
  75
  76 * node2
  77
  78 ----
  79 auto eno1
  80 iface eno1 inet manual
  81         mtu 1550
  82
  83 auto vmbr0
  84 iface vmbr0 inet static
  85         address 192.168.0.2
  86         netmask  255.255.255.0
  87         bridge_ports eno1
  88         bridge_stp off
  89         bridge_fd 0
  90
  91 auto vxlan2
  92 iface vxlan2 inet manual
  93         vxlan-svcnodeip 225.20.1.1
  94         vxlan-physdev eno1
  95
  96 auto vmbr2
  97 iface vmbr2 inet manual
  98         bridge_ports vxlan2
  99         bridge_stp off
 100         bridge_fd 0
 101
 102
 103 auto vxlan3
 104 iface vxlan3 inet manual
 105         vxlan-svcnodeip 225.20.1.1
 106         vxlan-physdev eno1
 107
 108 auto vmbr3
 109 iface vmbr3 inet manual
 110         bridge_ports vxlan3
 111         bridge_stp off
 112         bridge_fd 0
 113 ----
 114
 115
 116 * node3
 117
 118 ----
 119 auto eno1
 120 iface eno1 inet manual
 121         mtu 1550
 122
 123 auto vmbr0
 124 iface vmbr0 inet static
 125         address 192.168.0.3
 126         netmask  255.255.255.0
 127         bridge_ports eno1
 128         bridge_stp off
 129         bridge_fd 0
 130
 131 auto vxlan2
 132 iface vxlan2 inet manual
 133         vxlan-svcnodeip 225.20.1.1
 134         vxlan-physdev eno1
 135
 136 auto vmbr2
 137 iface vmbr2 inet manual
 138         bridge_ports vxlan2
 139         bridge_stp off
 140         bridge_fd 0
 141
 142
 143 auto vxlan3
 144 iface vxlan3 inet manual
 145         vxlan-svcnodeip 225.20.1.1
 146         vxlan-physdev eno1
 147
 148 auto vmbr3
 149 iface vmbr3 inet manual
 150         bridge_ports vxlan3
 151         bridge_stp off
 152         bridge_fd 0
 153 ----
 154
 155
 156 unicast mode
 157 ^^^^^^^^^^^^
 158
 159 We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs.
 160 The VXLAN is defined without a remote multicast group.
 161 Instead, all the remote VTEPs are associated with the all-zero address:
 162 a BUM frame will be duplicated to all these destinations.
 163 The VXLAN device will still learn remote addresses automatically using source-address learning.
 164
 165 * node1
 166
 167 ----
 168 auto eno1
 169 iface eno1 inet manual
 170         mtu 1550
 171
 172 auto vmbr0
 173 iface vmbr0 inet static
 174         address 192.168.0.1
 175         netmask  255.255.255.0
 176         bridge_ports eno1
 177         bridge_stp off
 178         bridge_fd 0
 179
 180
 181 auto vxlan2
 182 iface vxlan2 inet manual
 183         vxlan_remoteip 192.168.0.2
 184         vxlan_remoteip 192.168.0.3
 185
 186
 187 auto vmbr2
 188 iface vmbr2 inet manual
 189         bridge_ports vxlan2
 190         bridge_stp off
 191         bridge_fd 0
 192
 193
 194 auto vxlan3
 195 iface vxlan2 inet manual
 196         vxlan_remoteip 192.168.0.2
 197         vxlan_remoteip 192.168.0.3
 198
 199
 200 auto vmbr3
 201 iface vmbr3 inet manual
 202         bridge_ports vxlan3
 203         bridge_stp off
 204         bridge_fd 0
 205 ----
 206
 207
 208 * node2
 209
 210 ----
 211 auto eno1
 212 iface eno1 inet manual
 213         mtu 1550
 214
 215 auto vmbr0
 216 iface vmbr0 inet static
 217         address 192.168.0.2
 218         netmask  255.255.255.0
 219         bridge_ports eno1
 220         bridge_stp off
 221         bridge_fd 0
 222
 223 auto vxlan2
 224 iface vxlan2 inet manual
 225         vxlan_remoteip 192.168.0.1
 226         vxlan_remoteip 192.168.0.3
 227
 228
 229
 230 auto vmbr2
 231 iface vmbr2 inet manual
 232         bridge_ports vxlan2
 233         bridge_stp off
 234         bridge_fd 0
 235
 236 auto vxlan3
 237 iface vxlan2 inet manual
 238         vxlan_remoteip 192.168.0.1
 239         vxlan_remoteip 192.168.0.3
 240
 241
 242 auto vmbr3
 243 iface vmbr3 inet manual
 244         bridge_ports vxlan3
 245         bridge_stp off
 246         bridge_fd 0
 247 ----
 248
 249
 250 * node3
 251
 252 ----
 253 auto eno1
 254 iface eno1 inet manual
 255         mtu 1550
 256
 257 auto vmbr0
 258 iface vmbr0 inet static
 259         address 192.168.0.3
 260         netmask  255.255.255.0
 261         bridge_ports eno1
 262         bridge_stp off
 263         bridge_fd 0
 264
 265 auto vxlan2
 266 iface vxlan2 inet manual
 267         vxlan_remoteip 192.168.0.2
 268         vxlan_remoteip 192.168.0.3
 269
 270
 271
 272 auto vmbr2
 273 iface vmbr2 inet manual
 274         bridge_ports vxlan2
 275         bridge_stp off
 276         bridge_fd 0
 277
 278 auto vxlan3
 279 iface vxlan2 inet manual
 280         vxlan_remoteip 192.168.0.2
 281         vxlan_remoteip 192.168.0.3
 282
 283
 284 auto vmbr3
 285 iface vmbr3 inet manual
 286         bridge_ports vxlan3
 287         bridge_stp off
 288         bridge_fd 0
 289 ----
 290
 291
 292 bgp-evpn
 293 ^^^^^^^^
 294
 295 VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
 296 VTEPs have the ability to suppress ARP flooding over VXLAN tunnels.
 297
 298 The control plane used here is FRR, a bgp routing software.
 299 Each node in the proxmox cluster peer with each others nodes.
 300 For bigger networks, or multiple proxmox clusters,
 301 it's possible to use external bgp route reflector servers.
 302
 303 * node1
 304
 305 ----
 306 auto eno1
 307 iface eno1 inet manual
 308         mtu 1550
 309
 310 auto vmbr0
 311 iface vmbr0 inet static
 312         address 192.168.0.1
 313         netmask 255.255.255.0
 314         bridge_ports eno1
 315         bridge_stp off
 316         bridge_fd 0
 317
 318 auto vxlan2
 319 iface vxlan2 inet manual
 320         vxlan-local-tunnelip 192.168.0.1
 321         bridge-learning off
 322         bridge-arp-nd-suppress on
 323         bridge-unicast-flood off
 324         bridge-multicast-flood off
 325
 326
 327 auto vmbr2
 328 iface vmbr2 inet manual
 329         bridge_ports vxlan2
 330         bridge_stp off
 331         bridge_fd 0
 332
 333
 334 auto vxlan3
 335 iface vxlan3 inet manual
 336         vxlan-local-tunnelip 192.168.0.1
 337         bridge-learning off
 338         bridge-arp-nd-suppress on
 339         bridge-unicast-flood off
 340         bridge-multicast-flood off
 341
 342
 343 auto vmbr3
 344 iface vmbr3 inet manual
 345         bridge_ports vxlan3
 346         bridge_stp off
 347         bridge_fd 0
 348 ----
 349
 350
 351 /etc/frr/frr.conf
 352
 353 ----
 354 router bgp 1234
 355  no bgp default ipv4-unicast
 356  coalesce-time 1000
 357  neighbor 192.168.0.2 remote-as 1234
 358  neighbor 192.168.0.3 remote-as 1234
 359  !
 360  address-family l2vpn evpn
 361   neighbor 192.168.0.2 activate
 362   neighbor 192.168.0.3 activate
 363   advertise-all-vni
 364  exit-address-family
 365 !
 366 line vty
 367 !
 368 ----
 369
 370
 371 * node2
 372
 373 ----
 374 auto eno1
 375 iface eno1 inet manual
 376         mtu 1550
 377
 378 auto vmbr0
 379 iface vmbr0 inet static
 380         address 192.168.0.2
 381         netmask 255.255.255.0
 382         bridge_ports eno1
 383         bridge_stp off
 384         bridge_fd 0
 385
 386 auto vxlan2
 387 iface vxlan2 inet manual
 388         vxlan-local-tunnelip 192.168.0.2
 389         bridge-learning off
 390         bridge-arp-nd-suppress on
 391         bridge-unicast-flood off
 392         bridge-multicast-flood off
 393
 394
 395 auto vmbr2
 396 iface vmbr2 inet manual
 397         bridge_ports vxlan2
 398         bridge_stp off
 399         bridge_fd 0
 400
 401 auto vxlan3
 402 iface vxlan3 inet manual
 403         vxlan-local-tunnelip 192.168.0.2
 404         bridge-learning off
 405         bridge-arp-nd-suppress on
 406         bridge-unicast-flood off
 407         bridge-multicast-flood off
 408
 409
 410 auto vmbr3
 411 iface vmbr3 inet manual
 412         bridge_ports vxlan3
 413         bridge_stp off
 414         bridge_fd 0
 415 ----
 416
 417
 418 /etc/frr/frr.conf
 419
 420 ----
 421 router bgp 1234
 422  no bgp default ipv4-unicast
 423  coalesce-time 1000
 424  neighbor 192.168.0.1 remote-as 1234
 425  neighbor 192.168.0.3 remote-as 1234
 426  !
 427  address-family l2vpn evpn
 428   neighbor 192.168.0.1 activate
 429   neighbor 192.168.0.3 activate
 430   advertise-all-vni
 431  exit-address-family
 432 !
 433 line vty
 434 !
 435 ----
 436
 437
 438 * node3
 439
 440 ----
 441 auto eno1
 442 iface eno1 inet manual
 443         mtu 1550
 444
 445 auto vmbr0
 446 iface vmbr0 inet static
 447         address 192.168.0.2
 448         netmask 255.255.255.0
 449         bridge_ports eno1
 450         bridge_stp off
 451         bridge_fd 0
 452
 453 auto vxlan2
 454 iface vxlan2 inet manual
 455         vxlan-local-tunnelip 192.168.0.3
 456         bridge-learning off
 457         bridge-arp-nd-suppress on
 458         bridge-unicast-flood off
 459         bridge-multicast-flood off
 460
 461
 462 auto vmbr2
 463 iface vmbr2 inet manual
 464         bridge_ports vxlan2
 465         bridge_stp off
 466         bridge_fd 0
 467
 468 auto vxlan3
 469 iface vxlan3 inet manual
 470         vxlan-local-tunnelip 192.168.0.3
 471         bridge-learning off
 472         bridge-arp-nd-suppress on
 473         bridge-unicast-flood off
 474         bridge-multicast-flood off
 475
 476
 477 auto vmbr3
 478 iface vmbr3 inet manual
 479         bridge_ports vxlan3
 480         bridge_stp off
 481         bridge_fd 0
 482 ----
 483
 484
 485 /etc/frr/frr.conf
 486
 487
 488 ----
 489 router bgp 1234
 490  no bgp default ipv4-unicast
 491  coalesce-time 1000
 492  neighbor 192.168.0.1 remote-as 1234
 493  neighbor 192.168.0.2 remote-as 1234
 494  !
 495  address-family l2vpn evpn
 496   neighbor 192.168.0.1 activate
 497   neighbor 192.168.0.2 activate
 498   advertise-all-vni
 499  exit-address-family
 500 !
 501 line vty
 502 !
 503 ----
 504
 505 VXLAN layer3 routing with anycast gateway
 506 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 507
 508 With this need, each vmbr bridge will be the gateway for the vm.
 509 Same vmbr on different node, will have same ip address and same mac address,
 510 to have working vm live migration and no network disruption.
 511
 512 VXLAN layer3 routing only work with FRR and non-aware bridge.
 513 (vlan aware bridge support is buggy currently).
 514
 515 asymmetric model
 516 ^^^^^^^^^^^^^^^^
 517
 518 This is the simplest mode. To get it work, all vxlan need to be defined on all nodes.
 519
 520 The asymmetric model allows routing and bridging on the VXLAN tunnel ingress,
 521 but only bridging on the egress.
 522 This results in bi-directional VXLAN traffic traveling on different VNIs
 523 in each direction (always the destination VNI) across the routed infrastructure.
 524
 525 image::images/vxlan-l3-asymmetric.svg["vxlan l3 asymmetric",align="center"]
 526
 527
 528 sysctl.conf tuning
 529
 530 ----
 531 #enable routing
 532 net.ipv4.ip_forward=1
 533 net.ipv6.conf.all.forwarding=1
 534 ----
 535
 536 * node1
 537
 538 ----
 539 auto eno1
 540 iface eno1 inet manual
 541         mtu 1550
 542
 543 auto vmbr0
 544 iface vmbr0 inet static
 545          address 192.168.0.1
 546          netmask  255.255.255.0
 547          bridge_ports eno1
 548          bridge_stp off
 549          bridge_fd 0
 550
 551 auto vxlan2
 552 iface vxlan2 inet manual
 553         vxlan-local-tunnelip 192.168.0.1
 554         bridge-learning off
 555         bridge-arp-nd-suppress on
 556         bridge-unicast-flood off
 557         bridge-multicast-flood off
 558
 559
 560 auto vmbr2
 561 iface vmbr2 inet static
 562         address 10.0.2.254
 563         netmask 255.255.255.0
 564         hwaddress 44:39:39:FF:40:94
 565         bridge_ports vxlan2
 566         bridge_stp off
 567         bridge_fd 0
 568
 569
 570 auto vxlan3
 571 iface vxlan3 inet manual
 572         vxlan-local-tunnelip 192.168.0.1
 573         bridge-learning off
 574         bridge-arp-nd-suppress on
 575         bridge-unicast-flood off
 576         bridge-multicast-flood off
 577
 578
 579 auto vmbr3
 580 iface vmbr3 inet static
 581         address 10.0.3.254
 582         netmask 255.255.255.0
 583         hwaddress 44:39:39:FF:40:94
 584         bridge_ports vxlan3
 585         bridge_stp off
 586         bridge_fd 0
 587 ----
 588
 589
 590 frr.conf
 591
 592 ----
 593 router bgp 1234
 594  bgp router-id 192.168.0.1
 595  no bgp default ipv4-unicast
 596  coalesce-time 1000
 597  neighbor 192.168.0.2 remote-as 1234
 598  neighbor 192.168.0.3 remote-as 1234
 599  !
 600  address-family l2vpn evpn
 601   neighbor 192.168.0.2 activate
 602   neighbor 192.168.0.3 activate
 603   advertise-all-vni
 604  exit-address-family
 605 !
 606 line vty
 607 !
 608 ----
 609
 610
 611 * node2
 612
 613 ----
 614 auto eno1
 615 iface eno1 inet manual
 616         mtu 1550
 617
 618 auto vmbr0
 619 iface vmbr0 inet static
 620          address 192.168.0.2
 621          netmask  255.255.255.0
 622          bridge_ports eno1
 623          bridge_stp off
 624          bridge_fd 0
 625
 626 auto vxlan2
 627 iface vxlan2 inet manual
 628         vxlan-local-tunnelip 192.168.0.2
 629         bridge-learning off
 630         bridge-arp-nd-suppress on
 631         bridge-unicast-flood off
 632         bridge-multicast-flood off
 633
 634
 635 auto vmbr2
 636 iface vmbr2 inet static
 637         address 10.0.2.254
 638         netmask 255.255.255.0
 639         hwaddress 44:39:39:FF:40:94
 640         bridge_ports vxlan2
 641         bridge_stp off
 642         bridge_fd 0
 643
 644
 645 auto vxlan3
 646 iface vxlan3 inet manual
 647         vxlan-local-tunnelip 192.168.0.2
 648         bridge-learning off
 649         bridge-arp-nd-suppress on
 650         bridge-unicast-flood off
 651         bridge-multicast-flood off
 652
 653
 654 auto vmbr3
 655 iface vmbr3 inet static
 656         address 10.0.3.254
 657         netmask 255.255.255.0
 658         hwaddress 44:39:39:FF:40:94
 659         bridge_ports vxlan3
 660         bridge_stp off
 661         bridge_fd 0
 662 ----
 663
 664
 665 frr.conf
 666
 667 ----
 668 router bgp 1234
 669  bgp router-id 192.168.0.2
 670  no bgp default ipv4-unicast
 671  coalesce-time 1000
 672  neighbor 192.168.0.1 remote-as 1234
 673  neighbor 192.168.0.3 remote-as 1234
 674  !
 675  address-family l2vpn evpn
 676   neighbor 192.168.0.1 activate
 677   neighbor 192.168.0.3 activate
 678   advertise-all-vni
 679  exit-address-family
 680 !
 681 line vty
 682 !
 683 ----
 684
 685
 686 * node3
 687
 688 ----
 689 auto eno1
 690 iface eno1 inet manual
 691         mtu 1550
 692
 693 auto vmbr0
 694 iface vmbr0 inet static
 695          address 192.168.0.3
 696          netmask  255.255.255.0
 697          bridge_ports eno1
 698          bridge_stp off
 699          bridge_fd 0
 700
 701 auto vxlan2
 702 iface vxlan2 inet manual
 703         vxlan-local-tunnelip 192.168.0.3
 704         bridge-learning off
 705         bridge-arp-nd-suppress on
 706         bridge-unicast-flood off
 707         bridge-multicast-flood off
 708
 709
 710 auto vmbr2
 711 iface vmbr2 inet static
 712         address 10.0.2.254
 713         netmask 255.255.255.0
 714         hwaddress 44:39:39:FF:40:94
 715         bridge_ports vxlan2
 716         bridge_stp off
 717         bridge_fd 0
 718
 719
 720 auto vxlan3
 721 iface vxlan3 inet manual
 722         vxlan-local-tunnelip 192.168.0.3
 723         bridge-learning off
 724         bridge-arp-nd-suppress on
 725         bridge-unicast-flood off
 726         bridge-multicast-flood off
 727
 728
 729 auto vmbr3
 730 iface vmbr3 inet static
 731         address 10.0.3.254
 732         netmask 255.255.255.0
 733         hwaddress 44:39:39:FF:40:94
 734         bridge_ports vxlan3
 735         bridge_stp off
 736         bridge_fd 0
 737 ----
 738
 739
 740 frr.conf
 741
 742 ----
 743 router bgp 1234
 744  bgp router-id 192.168.0.3
 745  no bgp default ipv4-unicast
 746  coalesce-time 1000
 747  neighbor 192.168.0.1 remote-as 1234
 748  neighbor 192.168.0.2 remote-as 1234
 749  !
 750  address-family l2vpn evpn
 751   neighbor 192.168.0.1 activate
 752   neighbor 192.168.0.2 activate
 753   advertise-all-vni
 754  exit-address-family
 755 !
 756 line vty
 757 !
 758 ----
 759
 760
 761 symmetric model
 762 ^^^^^^^^^^^^^^^
 763
 764 With this model, you don't need to have all vxlan on all nodes.
 765 This model will also be needed to route traffic to an external router.
 766
 767 The symmetric model routes and bridges on both the ingress and the egress leafs.
 768 This results in bi-directional traffic being able to travel on the same VNI, hence the symmetric name.
 769 However, a new specialty transit VNI is used for all routed VXLAN traffic, called the L3VNI.
 770 All traffic that needs to be routed will be routed onto the L3VNI, tunneled across the layer 3 Infrastructure,
 771 routed off the L3VNI to the appropriate VLAN and ultimately bridged to the destination.
 772
 773 A vrf is needed for the L3VNI, so all vmbr bridge need to be in the vrf if they want to be able to reach each others.
 774
 775 image::images/vxlan-l3-symmetric.svg["vxlan l3 symmetric",align="center"]
 776
 777 sysctl.conf tuning
 778
 779 ----
 780 #enable routing
 781 net.ipv4.ip_forward=1
 782 net.ipv6.conf.all.forwarding=1
 783 #disable reverse path filtering
 784 net.ipv4.conf.default.rp_filter=0
 785 net.ipv4.conf.all.rp_filter=0
 786 ----
 787
 788 * node1
 789
 790 ----
 791 auto vrf1
 792 iface vrf1
 793     vrf-table auto
 794
 795 auto eno1
 796 iface eno1 inet manual
 797         mtu 1550
 798
 799 auto vmbr0
 800 iface vmbr0 inet static
 801          address 192.168.0.1
 802          netmask  255.255.255.0
 803          bridge_ports eno1
 804          bridge_stp off
 805          bridge_fd 0
 806
 807 auto vxlan2
 808 iface vxlan2 inet manual
 809         vxlan-local-tunnelip 192.168.0.1
 810         bridge-learning off
 811         bridge-arp-nd-suppress on
 812         bridge-unicast-flood off
 813         bridge-multicast-flood off
 814
 815 auto vmbr2
 816 iface vmbr2 inet static
 817         bridge_ports vxlan2
 818         bridge_stp off
 819         bridge_fd 0
 820         address 10.0.2.254
 821         netmask 255.255.255.0
 822         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 823         vrf vrf1
 824
 825 auto vxlan3
 826 iface vxlan3 inet manual
 827         vxlan-local-tunnelip 192.168.0.1
 828         bridge-learning off
 829         bridge-arp-nd-suppress on
 830         bridge-unicast-flood off
 831         bridge-multicast-flood off
 832
 833 auto vmbr3
 834 iface vmbr3 inet static
 835         bridge_ports vxlan3
 836         bridge_stp off
 837         bridge_fd 0
 838         address 10.0.3.254
 839         netmask 255.255.255.0
 840         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 841         vrf vrf1
 842
 843 #interconnect vxlan-vfr l3vni
 844 auto vxlan4000
 845 iface vxlan4000 inet manual
 846         vxlan-local-tunnelip 192.168.0.1
 847         bridge-learning off
 848         bridge-arp-nd-suppress on
 849         bridge-unicast-flood off
 850         bridge-multicast-flood off
 851
 852
 853 auto vmbr4000
 854 iface vmbr4000 inet manual
 855         bridge_ports vxlan4000
 856         bridge_stp off
 857         bridge_fd 0
 858         hwaddress 44:39:39:FF:40:90  #must be different on each node
 859         vrf vrf1
 860 ----
 861
 862 frr.conf
 863
 864 ----
 865 vrf vrf1
 866  vni 4000
 867 !
 868 router bgp 1234
 869  bgp router-id 192.168.0.1
 870  no bgp default ipv4-unicast
 871  coalesce-time 1000
 872  neighbor 192.168.0.2 remote-as 1234
 873  neighbor 192.168.0.3 remote-as 1234
 874  !
 875  address-family l2vpn evpn
 876   neighbor 192.168.0.2 activate
 877   neighbor 192.168.0.3 activate
 878   advertise-all-vni
 879  exit-address-family
 880 !
 881 router bgp 1234 vrf vrf1
 882 !
 883  bgp router-id 192.168.0.1
 884  !
 885  address-family ipv4 unicast
 886   redistribute connected
 887  exit-address-family
 888  !
 889  address-family l2vpn evpn
 890   advertise ipv4 unicast
 891  exit-address-family
 892 !
 893 line vty
 894 !
 895 ----
 896
 897
 898 * node2
 899
 900 ----
 901 auto vrf1
 902 iface vrf1
 903     vrf-table auto
 904
 905 auto eno1
 906 iface eno1 inet manual
 907         mtu 1550
 908
 909 auto vmbr0
 910 iface vmbr0 inet static
 911          address 192.168.0.2
 912          netmask  255.255.255.0
 913          bridge_ports eno1
 914          bridge_stp off
 915          bridge_fd 0
 916
 917 auto vxlan2
 918 iface vxlan2 inet manual
 919         vxlan-local-tunnelip 192.168.0.2
 920         bridge-learning off
 921         bridge-arp-nd-suppress on
 922         bridge-unicast-flood off
 923         bridge-multicast-flood off
 924
 925 auto vmbr2
 926 iface vmbr2 inet static
 927         bridge_ports vxlan2
 928         bridge_stp off
 929         bridge_fd 0
 930         address 10.0.2.254
 931         netmask 255.255.255.0
 932         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
 933         vrf vrf1
 934
 935 auto vxlan3
 936 iface vxlan3 inet manual
 937         vxlan-local-tunnelip 192.168.0.2
 938         bridge-learning off
 939         bridge-arp-nd-suppress on
 940         bridge-unicast-flood off
 941         bridge-multicast-flood off
 942
 943 auto vmbr3
 944 iface vmbr3 inet static
 945         bridge_ports vxlan3
 946         bridge_stp off
 947         bridge_fd 0
 948         address 10.0.3.254
 949         netmask 255.255.255.0
 950         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
 951         vrf vrf1
 952
 953 #interconnect vxlan-vfr l3vni
 954 auto vxlan4000
 955 iface vxlan4000 inet manual
 956         vxlan-local-tunnelip 192.168.0.2
 957         bridge-learning off
 958         bridge-arp-nd-suppress on
 959         bridge-unicast-flood off
 960         bridge-multicast-flood off
 961
 962
 963 auto vmbr4000
 964 iface vmbr4000 inet manual
 965         bridge_ports vxlan4000
 966         bridge_stp off
 967         bridge_fd 0
 968         hwaddress 44:39:39:FF:40:91  #must be different on each node
 969         vrf vrf1
 970 ----
 971
 972
 973 frr.conf
 974
 975 ----
 976 vrf vrf1
 977  vni 4000
 978 !
 979 router bgp 1234
 980  bgp router-id 192.168.0.2
 981  no bgp default ipv4-unicast
 982  coalesce-time 1000
 983  neighbor 192.168.0.1 remote-as 1234
 984  neighbor 192.168.0.3 remote-as 1234
 985  !
 986  address-family l2vpn evpn
 987   neighbor 192.168.0.1 activate
 988   neighbor 192.168.0.3 activate
 989   advertise-all-vni
 990  exit-address-family
 991 !
 992 router bgp 1234 vrf vrf1
 993 !
 994  bgp router-id 192.168.0.2
 995  !
 996  address-family ipv4 unicast
 997   redistribute connected
 998  exit-address-family
 999  !
1000  address-family l2vpn evpn
1001   advertise ipv4 unicast
1002  exit-address-family
1003 !
1004 line vty
1005 !
1006 ----
1007
1008
1009 * node3
1010
1011 ----
1012 auto vrf1
1013 iface vrf1
1014     vrf-table auto
1015
1016 auto eno1
1017 iface eno1 inet manual
1018         mtu 1550
1019
1020 auto vmbr0
1021 iface vmbr0 inet static
1022          address 192.168.0.3
1023          netmask  255.255.255.0
1024          bridge_ports eno1
1025          bridge_stp off
1026          bridge_fd 0
1027
1028 auto vxlan2
1029 iface vxlan2 inet manual
1030         vxlan-local-tunnelip 192.168.0.3
1031         bridge-learning off
1032         bridge-arp-nd-suppress on
1033         bridge-unicast-flood off
1034         bridge-multicast-flood off
1035
1036 auto vmbr2
1037 iface vmbr2 inet static
1038         bridge_ports vxlan2
1039         bridge_stp off
1040         bridge_fd 0
1041         address 10.0.2.254
1042         netmask 255.255.255.0
1043         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1044         vrf vrf1
1045
1046 auto vxlan3
1047 iface vxlan3 inet manual
1048         vxlan-local-tunnelip 192.168.0.3
1049         bridge-learning off
1050         bridge-arp-nd-suppress on
1051         bridge-unicast-flood off
1052         bridge-multicast-flood off
1053
1054 auto vmbr3
1055 iface vmbr3 inet static
1056         bridge_ports vxlan3
1057         bridge_stp off
1058         bridge_fd 0
1059         address 10.0.3.254
1060         netmask 255.255.255.0
1061         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1062         vrf vrf1
1063
1064 #interconnect vxlan-vfr l3vni
1065 auto vxlan4000
1066 iface vxlan4000 inet manual
1067         vxlan-local-tunnelip 192.168.0.3
1068         bridge-learning off
1069         bridge-arp-nd-suppress on
1070         bridge-unicast-flood off
1071         bridge-multicast-flood off
1072
1073
1074 auto vmbr4000
1075 iface vmbr4000 inet manual
1076         bridge_ports vxlan4000
1077         bridge_stp off
1078         bridge_fd 0
1079         hwaddress 44:39:39:FF:40:92  #must be different on each node
1080         vrf vrf1
1081 ----
1082
1083
1084 frr.conf
1085
1086 ----
1087 vrf vrf1
1088  vni 4000
1089 !
1090 router bgp 1234
1091  bgp router-id 192.168.0.3
1092  no bgp default ipv4-unicast
1093  coalesce-time 1000
1094  neighbor 192.168.0.1 remote-as 1234
1095  neighbor 192.168.0.2 remote-as 1234
1096  !
1097  address-family l2vpn evpn
1098   neighbor 192.168.0.1 activate
1099   neighbor 192.168.0.2 activate
1100   advertise-all-vni
1101  exit-address-family
1102 !
1103 router bgp 1234 vrf vrf1
1104 !
1105  bgp router-id 192.168.0.3
1106  !
1107  address-family ipv4 unicast
1108   redistribute connected
1109  exit-address-family
1110  !
1111  address-family l2vpn evpn
1112   advertise ipv4 unicast
1113  exit-address-family
1114 !
1115 line vty
1116 !
1117 ----
1118
1119 VXLAN layer3 routing with anycast gateway + routing to outside with external router
1120 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1121 Routing to outside need the symmetric model.
1122
1123 1 gateway node
1124 ^^^^^^^^^^^^^^
1125 In this example, we'll use only 1 proxmox node as exit gateway. (node1)
1126 This node have a simple default gw in the vrf to the external router (no bgp between router and node1)
1127 and announce this default gw to other proxmox nodes.
1128
1129
1130 *node1
1131
1132 ----
1133 auto vrf1
1134 iface vrf1
1135     vrf-table auto
1136
1137 auto eno1
1138 iface eno1 inet manual
1139         mtu 1550
1140
1141 auto vmbr0
1142 iface vmbr0 inet static
1143          address 192.168.0.1
1144          netmask  255.255.255.0
1145          bridge_ports eno1
1146          bridge_stp off
1147          bridge_fd 0
1148
1149 auto eno2
1150 iface eno2
1151         address 172.16.0.1
1152         netmask 255.255.255.0
1153         vrf vrf1
1154         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1155         #if you have multiple external routers, you can use ecmp balancing
1156         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1157
1158 auto vxlan2
1159 iface vxlan2 inet manual
1160         vxlan-local-tunnelip 192.168.0.1
1161         bridge-learning off
1162         bridge-arp-nd-suppress on
1163         bridge-unicast-flood off
1164         bridge-multicast-flood off
1165
1166 auto vmbr2
1167 iface vmbr2 inet static
1168         bridge_ports vxlan2
1169         bridge_stp off
1170         bridge_fd 0
1171         address 10.0.2.254
1172         netmask 255.255.255.0
1173         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1174         vrf vrf1
1175
1176 auto vxlan3
1177 iface vxlan3 inet manual
1178         vxlan-local-tunnelip 192.168.0.1
1179         bridge-learning off
1180         bridge-arp-nd-suppress on
1181         bridge-unicast-flood off
1182         bridge-multicast-flood off
1183
1184 auto vmbr3
1185 iface vmbr3 inet static
1186         bridge_ports vxlan3
1187         bridge_stp off
1188         bridge_fd 0
1189         address 10.0.3.254
1190         netmask 255.255.255.0
1191         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1192         vrf vrf1
1193
1194 #interconnect vxlan-vfr l3vni
1195 auto vxlan4000
1196 iface vxlan4000 inet manual
1197         vxlan-local-tunnelip 192.168.0.1
1198         bridge-learning off
1199         bridge-arp-nd-suppress on
1200         bridge-unicast-flood off
1201         bridge-multicast-flood off
1202
1203 auto vmbr4000
1204 iface vmbr4000 inet manual
1205         bridge_ports vxlan4000
1206         bridge_stp off
1207         bridge_fd 0
1208         hwaddress 44:39:39:FF:40:90  #must be different on each node
1209         vrf vrf1
1210 ----
1211
1212
1213 frr.conf
1214
1215 ----
1216 vrf vrf1
1217  vni 4000
1218 !
1219 router bgp 1234
1220  bgp router-id 192.168.0.1
1221  no bgp default ipv4-unicast
1222  coalesce-time 1000
1223  neighbor 192.168.0.2 remote-as 1234
1224  neighbor 192.168.0.3 remote-as 1234
1225  !
1226  address-family l2vpn evpn
1227   neighbor 192.168.0.2 activate
1228   neighbor 192.168.0.3 activate
1229   advertise-all-vni
1230  exit-address-family
1231 !
1232 router bgp 1234 vrf vrf1
1233 !
1234  bgp router-id 172.16.0.1
1235  !
1236  address-family ipv4 unicast
1237   redistribute connected
1238   redistribute kernel !announce your default gw to all nodes
1239  exit-address-family
1240  !
1241  address-family l2vpn evpn
1242   advertise ipv4 unicast
1243  exit-address-family
1244 !
1245 line vty
1246 !
1247 ----
1248
1249
1250 * node2
1251
1252 ----
1253 auto vrf1
1254 iface vrf1
1255     vrf-table auto
1256
1257 auto eno1
1258 iface eno1 inet manual
1259         mtu 1550
1260
1261 auto vmbr0
1262 iface vmbr0 inet static
1263          address 192.168.0.2
1264          netmask  255.255.255.0
1265          bridge_ports eno1
1266          bridge_stp off
1267          bridge_fd 0
1268
1269 auto vxlan2
1270 iface vxlan2 inet manual
1271         vxlan-local-tunnelip 192.168.0.2
1272         bridge-learning off
1273         bridge-arp-nd-suppress on
1274         bridge-unicast-flood off
1275         bridge-multicast-flood off
1276
1277 auto vmbr2
1278 iface vmbr2 inet static
1279         bridge_ports vxlan2
1280         bridge_stp off
1281         bridge_fd 0
1282         address 10.0.2.254
1283         netmask 255.255.255.0
1284         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1285         vrf vrf1
1286
1287 auto vxlan3
1288 iface vxlan3 inet manual
1289         vxlan-local-tunnelip 192.168.0.2
1290         bridge-learning off
1291         bridge-arp-nd-suppress on
1292         bridge-unicast-flood off
1293         bridge-multicast-flood off
1294
1295 auto vmbr3
1296 iface vmbr3 inet static
1297         bridge_ports vxlan3
1298         bridge_stp off
1299         bridge_fd 0
1300         address 10.0.3.254
1301         netmask 255.255.255.0
1302         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1303         vrf vrf1
1304
1305 #interconnect vxlan-vfr l3vni
1306 auto vxlan4000
1307 iface vxlan4000 inet manual
1308         vxlan-local-tunnelip 192.168.0.2
1309         bridge-learning off
1310         bridge-arp-nd-suppress on
1311         bridge-unicast-flood off
1312         bridge-multicast-flood off
1313
1314
1315 auto vmbr4000
1316 iface vmbr4000 inet manual
1317         bridge_ports vxlan4000
1318         bridge_stp off
1319         bridge_fd 0
1320         hwaddress 44:39:39:FF:40:91  #must be different on each node
1321         vrf vrf1
1322 ----
1323
1324
1325 frr.conf
1326
1327 ----
1328 vrf vrf1
1329  vni 4000
1330 !
1331 router bgp 1234
1332  bgp router-id 192.168.0.2
1333  no bgp default ipv4-unicast
1334  coalesce-time 1000
1335  neighbor 192.168.0.1 remote-as 1234
1336  neighbor 192.168.0.3 remote-as 1234
1337  !
1338  address-family l2vpn evpn
1339   neighbor 192.168.0.1 activate
1340   neighbor 192.168.0.3 activate
1341   advertise-all-vni
1342  exit-address-family
1343 !
1344 router bgp 1234 vrf vrf1
1345 !
1346  bgp router-id 192.168.0.2
1347  !
1348  address-family ipv4 unicast
1349   redistribute connected
1350  exit-address-family
1351  !
1352  address-family l2vpn evpn
1353   advertise ipv4 unicast
1354  exit-address-family
1355 !
1356 line vty
1357 !
1358 ----
1359
1360
1361 * node3
1362
1363 ----
1364 auto vrf1
1365 iface vrf1
1366     vrf-table auto
1367
1368 auto eno1
1369 iface eno1 inet manual
1370         mtu 1550
1371
1372 auto vmbr0
1373 iface vmbr0 inet static
1374          address 192.168.0.3
1375          netmask  255.255.255.0
1376          bridge_ports eno1
1377          bridge_stp off
1378          bridge_fd 0
1379
1380 auto vxlan2
1381 iface vxlan2 inet manual
1382         vxlan-local-tunnelip 192.168.0.3
1383         bridge-learning off
1384         bridge-arp-nd-suppress on
1385         bridge-unicast-flood off
1386         bridge-multicast-flood off
1387
1388 auto vmbr2
1389 iface vmbr2 inet static
1390         bridge_ports vxlan2
1391         bridge_stp off
1392         bridge_fd 0
1393         address 10.0.2.254
1394         netmask 255.255.255.0
1395         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1396         vrf vrf1
1397
1398 auto vxlan3
1399 iface vxlan3 inet manual
1400         vxlan-local-tunnelip 192.168.0.3
1401         bridge-learning off
1402         bridge-arp-nd-suppress on
1403         bridge-unicast-flood off
1404         bridge-multicast-flood off
1405
1406 auto vmbr3
1407 iface vmbr3 inet static
1408         bridge_ports vxlan3
1409         bridge_stp off
1410         bridge_fd 0
1411         address 10.0.3.254
1412         netmask 255.255.255.0
1413         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1414         vrf vrf1
1415
1416 #interconnect vxlan-vfr l3vni
1417 auto vxlan4000
1418 iface vxlan4000 inet manual
1419         vxlan-local-tunnelip 192.168.0.3
1420         bridge-learning off
1421         bridge-arp-nd-suppress on
1422         bridge-unicast-flood off
1423         bridge-multicast-flood off
1424
1425
1426 auto vmbr4000
1427 iface vmbr4000 inet manual
1428         bridge_ports vxlan4000
1429         bridge_stp off
1430         bridge_fd 0
1431         hwaddress 44:39:39:FF:40:92  #must be different on each node
1432         vrf vrf1
1433 ----
1434
1435
1436 frr.conf
1437
1438 ----
1439 vrf vrf1
1440  vni 4000
1441 !
1442 router bgp 1234
1443  bgp router-id 192.168.0.3
1444  no bgp default ipv4-unicast
1445  coalesce-time 1000
1446  neighbor 192.168.0.1 remote-as 1234
1447  neighbor 192.168.0.2 remote-as 1234
1448  !
1449  address-family l2vpn evpn
1450   neighbor 192.168.0.1 activate
1451   neighbor 192.168.0.2 activate
1452   advertise-all-vni
1453  exit-address-family
1454 !
1455 router bgp 1234 vrf vrf1
1456 !
1457  bgp router-id 192.168.0.3
1458  !
1459  address-family ipv4 unicast
1460   redistribute connected
1461  exit-address-family
1462  !
1463  address-family l2vpn evpn
1464   advertise ipv4 unicast
1465  exit-address-family
1466 !
1467 line vty
1468 !
1469 ----
1470
1471 multiple gateway nodes
1472 ^^^^^^^^^^^^^^^^^^^^^^
1473 In this example, all nodes will be used as exit gateway. (But you can use only 2 nodes if you want)
1474 All nodes have a simple default gw in the vrf to the external router (no bgp between router and node1)
1475 and announce this default gw.
1476 The external router have ecmp routes to all proxmox nodes.(balancing).
1477 If the router send the packet to a wrong node (vm is not on this node), this node will route through
1478 vxlan the packet to final destination.
1479
1480 *node1
1481
1482 ----
1483 auto vrf1
1484 iface vrf1
1485     vrf-table auto
1486
1487 auto eno1
1488 iface eno1 inet manual
1489         mtu 1550
1490
1491 auto vmbr0
1492 iface vmbr0 inet static
1493          address 192.168.0.1
1494          netmask  255.255.255.0
1495          bridge_ports eno1
1496          bridge_stp off
1497          bridge_fd 0
1498
1499 auto eno2
1500 iface eno2
1501         address 172.16.0.1
1502         netmask 255.255.255.0
1503         vrf vrf1
1504         mtu 1550
1505         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1506         #if you have multiple external routers, you can use ecmp balancing
1507         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1508
1509 auto vxlan2
1510 iface vxlan2 inet manual
1511         vxlan-local-tunnelip 192.168.0.1
1512         bridge-learning off
1513         bridge-arp-nd-suppress on
1514         bridge-unicast-flood off
1515         bridge-multicast-flood off
1516
1517 auto vmbr2
1518 iface vmbr2 inet static
1519         bridge_ports vxlan2
1520         bridge_stp off
1521         bridge_fd 0
1522         address 10.0.2.254
1523         netmask 255.255.255.0
1524         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1525         vrf vrf1
1526
1527 auto vxlan3
1528 iface vxlan3 inet manual
1529         vxlan-local-tunnelip 192.168.0.1
1530         bridge-learning off
1531         bridge-arp-nd-suppress on
1532         bridge-unicast-flood off
1533         bridge-multicast-flood off
1534
1535 auto vmbr3
1536 iface vmbr3 inet static
1537         bridge_ports vxlan3
1538         bridge_stp off
1539         bridge_fd 0
1540         address 10.0.3.254
1541         netmask 255.255.255.0
1542         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1543         vrf vrf1
1544
1545 #interconnect vxlan-vfr l3vni
1546 auto vxlan4000
1547 iface vxlan4000 inet manual
1548         vxlan-local-tunnelip 192.168.0.1
1549         bridge-learning off
1550         bridge-arp-nd-suppress on
1551         bridge-unicast-flood off
1552         bridge-multicast-flood off
1553
1554 auto vmbr4000
1555 iface vmbr4000 inet manual
1556         bridge_ports vxlan4000
1557         bridge_stp off
1558         bridge_fd 0
1559         hwaddress 44:39:39:FF:40:90  #must be different on each node
1560         vrf vrf1
1561 ----
1562
1563
1564 frr.conf
1565
1566 ----
1567 vrf vrf1
1568  vni 4000
1569 !
1570 router bgp 1234
1571  bgp router-id 192.168.0.1
1572  no bgp default ipv4-unicast
1573  coalesce-time 1000
1574  neighbor 192.168.0.2 remote-as 1234
1575  neighbor 192.168.0.3 remote-as 1234
1576  !
1577  address-family l2vpn evpn
1578   neighbor 192.168.0.2 activate
1579   neighbor 192.168.0.3 activate
1580   advertise-all-vni
1581  exit-address-family
1582 !
1583 router bgp 1234 vrf vrf1
1584 !
1585  bgp router-id 172.16.0.1
1586  !
1587  address-family ipv4 unicast
1588   redistribute connected
1589   redistribute kernel !announce your default gw to all nodes
1590  exit-address-family
1591  !
1592  address-family l2vpn evpn
1593   advertise ipv4 unicast
1594  exit-address-family
1595 !
1596 line vty
1597 !
1598 ----
1599
1600
1601 * node2
1602
1603 ----
1604 auto vrf1
1605 iface vrf1
1606     vrf-table auto
1607
1608 auto eno1
1609 iface eno1 inet manual
1610         mtu 1550
1611
1612 auto vmbr0
1613 iface vmbr0 inet static
1614          address 192.168.0.2
1615          netmask  255.255.255.0
1616          bridge_ports eno1
1617          bridge_stp off
1618          bridge_fd 0
1619
1620 auto eno2
1621 iface eno2
1622         address 172.16.0.3
1623         netmask 255.255.255.0
1624         vrf vrf1
1625         mtu 1550
1626         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1627         #if you have multiple external routers, you can use ecmp balancing
1628         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1629
1630 auto vxlan2
1631 iface vxlan2 inet manual
1632         vxlan-local-tunnelip 192.168.0.2
1633         bridge-learning off
1634         bridge-arp-nd-suppress on
1635         bridge-unicast-flood off
1636         bridge-multicast-flood off
1637
1638 auto vmbr2
1639 iface vmbr2 inet static
1640         bridge_ports vxlan2
1641         bridge_stp off
1642         bridge_fd 0
1643         address 10.0.2.254
1644         netmask 255.255.255.0
1645         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1646         vrf vrf1
1647
1648 auto vxlan3
1649 iface vxlan3 inet manual
1650         vxlan-local-tunnelip 192.168.0.2
1651         bridge-learning off
1652         bridge-arp-nd-suppress on
1653         bridge-unicast-flood off
1654         bridge-multicast-flood off
1655
1656 auto vmbr3
1657 iface vmbr3 inet static
1658         bridge_ports vxlan3
1659         bridge_stp off
1660         bridge_fd 0
1661         address 10.0.3.254
1662         netmask 255.255.255.0
1663         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1664         vrf vrf1
1665
1666 #interconnect vxlan-vfr l3vni
1667 auto vxlan4000
1668 iface vxlan4000 inet manual
1669         vxlan-local-tunnelip 192.168.0.2
1670         bridge-learning off
1671         bridge-arp-nd-suppress on
1672         bridge-unicast-flood off
1673         bridge-multicast-flood off
1674
1675
1676 auto vmbr4000
1677 iface vmbr4000 inet manual
1678         bridge_ports vxlan4000
1679         bridge_stp off
1680         bridge_fd 0
1681         hwaddress 44:39:39:FF:40:91  #must be different on each node
1682         vrf vrf1
1683 ----
1684
1685
1686 frr.conf
1687
1688 ----
1689 vrf vrf1
1690  vni 4000
1691 !
1692 router bgp 1234
1693  bgp router-id 192.168.0.2
1694  no bgp default ipv4-unicast
1695  coalesce-time 1000
1696  neighbor 192.168.0.1 remote-as 1234
1697  neighbor 192.168.0.3 remote-as 1234
1698  !
1699  address-family l2vpn evpn
1700   neighbor 192.168.0.1 activate
1701   neighbor 192.168.0.3 activate
1702   advertise-all-vni
1703  exit-address-family
1704 !
1705 router bgp 1234 vrf vrf1
1706 !
1707  bgp router-id 172.16.0.2
1708  !
1709  address-family ipv4 unicast
1710   redistribute connected
1711   redistribute kernel !announce your default gw to all nodes
1712  exit-address-family
1713  !
1714  address-family l2vpn evpn
1715   advertise ipv4 unicast
1716  exit-address-family
1717 !
1718 line vty
1719 !
1720 ----
1721
1722
1723 * node3
1724
1725 ----
1726 auto vrf1
1727 iface vrf1
1728     vrf-table auto
1729
1730 auto eno1
1731 iface eno1 inet manual
1732         mtu 1550
1733
1734 auto vmbr0
1735 iface vmbr0 inet static
1736          address 192.168.0.3
1737          netmask  255.255.255.0
1738          bridge_ports eno1
1739          bridge_stp off
1740          bridge_fd 0
1741
1742 auto eno2
1743 iface eno2
1744         address 172.16.0.3
1745         netmask 255.255.255.0
1746         vrf vrf1
1747         mtu 1550
1748         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1749         #if you have multiple external routers, you can use ecmp balancing
1750         #post-up route add default nexthop via 172.16.0.253 dev eno2 vrf vrf1 nexthop via 172.16.0.254 dev eno2 vrf vrf1
1751
1752 auto vxlan2
1753 iface vxlan2 inet manual
1754         vxlan-local-tunnelip 192.168.0.3
1755         bridge-learning off
1756         bridge-arp-nd-suppress on
1757         bridge-unicast-flood off
1758         bridge-multicast-flood off
1759
1760 auto vmbr2
1761 iface vmbr2 inet static
1762         bridge_ports vxlan2
1763         bridge_stp off
1764         bridge_fd 0
1765         address 10.0.2.254
1766         netmask 255.255.255.0
1767         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
1768         vrf vrf1
1769
1770 auto vxlan3
1771 iface vxlan3 inet manual
1772         vxlan-local-tunnelip 192.168.0.3
1773         bridge-learning off
1774         bridge-arp-nd-suppress on
1775         bridge-unicast-flood off
1776         bridge-multicast-flood off
1777
1778 auto vmbr3
1779 iface vmbr3 inet static
1780         bridge_ports vxlan3
1781         bridge_stp off
1782         bridge_fd 0
1783         address 10.0.3.254
1784         netmask 255.255.255.0
1785         hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
1786         vrf vrf1
1787
1788 #interconnect vxlan-vfr l3vni
1789 auto vxlan4000
1790 iface vxlan4000 inet manual
1791         vxlan-local-tunnelip 192.168.0.3
1792         bridge-learning off
1793         bridge-arp-nd-suppress on
1794         bridge-unicast-flood off
1795         bridge-multicast-flood off
1796
1797
1798 auto vmbr4000
1799 iface vmbr4000 inet manual
1800         bridge_ports vxlan4000
1801         bridge_stp off
1802         bridge_fd 0
1803         hwaddress 44:39:39:FF:40:92  #must be different on each node
1804         vrf vrf1
1805 ----
1806
1807
1808 frr.conf
1809
1810 ----
1811 vrf vrf1
1812  vni 4000
1813 !
1814 router bgp 1234
1815  bgp router-id 192.168.0.3
1816  no bgp default ipv4-unicast
1817  coalesce-time 1000
1818  neighbor 192.168.0.1 remote-as 1234
1819  neighbor 192.168.0.2 remote-as 1234
1820  !
1821  address-family l2vpn evpn
1822   neighbor 192.168.0.1 activate
1823   neighbor 192.168.0.2 activate
1824   advertise-all-vni
1825  exit-address-family
1826 !
1827 router bgp 1234 vrf vrf1
1828 !
1829  bgp router-id 172.16.0.3
1830  !
1831  address-family ipv4 unicast
1832   redistribute connected
1833   redistribute kernel !announce your default gw to all nodes
1834  exit-address-family
1835  !
1836  address-family l2vpn evpn
1837   advertise ipv4 unicast
1838  exit-address-family
1839 !
1840 line vty
1841 !
1842 ----
1843
1844 Note
1845 ^^^^
1846
1847 If your external router don't support ecmp to reach multiple proxmox nodes,
1848 you can setup an HA floating vip on proxmox nodes with vrrp
1849
1850 I this example, we will setup an floating 172.16.0.10 ip on node1 and node2.
1851 Node1 is the primary and failover to node2 in case of failure.
1852
1853
1854 * node1
1855
1856 ----
1857 auto eno2
1858 iface eno2
1859         address 172.16.0.1
1860         netmask 255.255.255.0
1861         vrf vrf1
1862         mtu 1550
1863         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1864         vrrp-id 1
1865         vrrp-priority 1
1866         vrrp-virtual-ip 172.16.0.10
1867 ----
1868
1869 * node2
1870
1871 ----
1872 auto eno2
1873 iface eno2
1874         address 172.16.0.2
1875         netmask 255.255.255.0
1876         mtu 1550
1877         vrf vrf1
1878         post-up ip route add default via 172.16.0.254 dev eno2 vrf vrf1
1879         vrrp-id 1
1880         vrrp-priority 2
1881         vrrp-virtual-ip 172.16.0.10
1882 ----
1883
1884