pvesdn.adoc

   1 [[chapter_pvesdn]]
   2 Software Defined Network
   3 ========================
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 The **S**oftware **D**efined **N**etwork (SDN) feature allows you to create
   9 virtual networks (VNets) at the datacenter level.
  10
  11 WARNING: SDN is currently an **experimental feature** in {pve}. This
  12 documentation for it is also still under development. Ask on our
  13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
  14
  15
  16 [[pvesdn_installation]]
  17 Installation
  18 ------------
  19
  20 To enable the experimental Software Defined Network (SDN) integration, you need
  21 to install the `libpve-network-perl` and `ifupdown2` packages on every node:
  22
  23 ----
  24 apt update
  25 apt install libpve-network-perl ifupdown2
  26 ----
  27
  28 NOTE: {pve} version 7 and above come installed with ifupdown2.
  29
  30 After this, you need to add the following line to the end of the
  31 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
  32 included and activated.
  33
  34 ----
  35 source /etc/network/interfaces.d/*
  36 ----
  37
  38
  39 Basic Overview
  40 --------------
  41
  42 The {pve} SDN allows for separation and fine-grained control of virtual guest
  43 networks, using flexible, software-controlled configurations.
  44
  45 Separation is managed through zones, where a zone is its own virtual separated
  46 network area. A 'VNet' is a type of a virtual network connected to a zone.
  47 Depending on which type or plugin the zone uses, it can behave differently and
  48 offer different features, advantages, and disadvantages. Normally, a 'VNet'
  49 appears as a common Linux bridge with either a VLAN or 'VXLAN' tag, however,
  50 some can also use layer 3 routing for control. 'VNets' are deployed locally on
  51 each node, after being configured from the cluster-wide datacenter SDN
  52 administration interface.
  53
  54
  55 Main Configuration
  56 ~~~~~~~~~~~~~~~~~~
  57
  58 Configuration is done at the datacenter (cluster-wide) level and is saved in
  59 files located in the shared configuration file system:
  60 `/etc/pve/sdn`
  61
  62 On the web-interface, SDN features 3 main sections:
  63
  64 * SDN: An overview of the SDN state
  65
  66 * Zones: Create and manage the virtually separated network zones
  67
  68 * VNets: Create virtual network bridges and manage subnets
  69
  70 In addition to this, the following options are offered:
  71
  72 * Controller: For controlling layer 3 routing in complex setups
  73
  74 * Subnets: Used to defined IP networks on VNets
  75
  76 * IPAM: Enables the use of external tools for IP address management (guest
  77   IPs)
  78
  79 * DNS: Define a DNS server API for registering virtual guests' hostname and IP
  80   addresses
  81
  82 [[pvesdn_config_main_sdn]]
  83
  84 SDN
  85 ~~~
  86
  87 This is the main status panel. Here you can see the deployment status of zones
  88 on different nodes.
  89
  90 The 'Apply' button is used to push and reload local configuration on all cluster
  91 nodes.
  92
  93
  94 [[pvesdn_local_deployment_monitoring]]
  95 Local Deployment Monitoring
  96 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  97
  98 After applying the configuration through the main SDN panel,
  99 the local network configuration is generated locally on each node in
 100 the file `/etc/network/interfaces.d/sdn`, and reloaded with ifupdown2.
 101
 102 You can monitor the status of local zones and VNets through the main tree.
 103
 104
 105 [[pvesdn_config_zone]]
 106 Zones
 107 -----
 108
 109 A zone defines a virtually separated network. Zones can be restricted to
 110 specific nodes and assigned permissions, in order to restrict users to a certain
 111 zone and its contained VNets.
 112
 113 Different technologies can be used for separation:
 114
 115 * VLAN: Virtual LANs are the classic method of subdividing a LAN
 116
 117 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
 118
 119 * VXLAN: Layer2 VXLAN
 120
 121 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
 122
 123 * EVPN (BGP EVPN): VXLAN using layer 3 border gateway protocol (BGP) routing
 124
 125 Common options
 126 ~~~~~~~~~~~~~~
 127
 128 The following options are available for all zone types:
 129
 130 nodes:: The nodes which the zone and associated VNets should be deployed on
 131
 132 ipam:: Optional. Use an IP Address Management (IPAM) tool to manage IPs in the
 133   zone.
 134
 135 dns:: Optional. DNS API server.
 136
 137 reversedns:: Optional. Reverse DNS API server.
 138
 139 dnszone:: Optional. DNS domain name. Used to register hostnames, such as
 140   `<hostname>.<domain>`. The DNS zone must already exist on the DNS server.
 141
 142
 143 [[pvesdn_zone_plugin_simple]]
 144 Simple Zones
 145 ~~~~~~~~~~~~
 146
 147 This is the simplest plugin. It will create an isolated VNet bridge.
 148 This bridge is not linked to a physical interface, and VM traffic is only
 149 local between the node(s).
 150 It can also be used in NAT or routed setups.
 151
 152 [[pvesdn_zone_plugin_vlan]]
 153 VLAN Zones
 154 ~~~~~~~~~~
 155
 156 This plugin reuses an existing local Linux or OVS bridge, and manages the VLANs
 157 on it. The benefit of using the SDN module is that you can create different
 158 zones with specific VNet VLAN tags, and restrict virtual machines to separated
 159 zones.
 160
 161 Specific `VLAN` configuration options:
 162
 163 bridge:: Reuse this local bridge or OVS switch, already configured on *each*
 164   local node.
 165
 166 [[pvesdn_zone_plugin_qinq]]
 167 QinQ Zones
 168 ~~~~~~~~~~
 169
 170 QinQ also known as VLAN stacking, wherein the first VLAN tag is defined for the
 171 zone (the 'service-vlan'), and the second VLAN tag is defined for the
 172 VNets.
 173
 174 NOTE: Your physical network switches must support stacked VLANs for this
 175 configuration!
 176
 177 Below are the configuration options specific to QinQ:
 178
 179 bridge:: A local, VLAN-aware bridge that is already configured on each local
 180   node
 181
 182 service vlan:: The main VLAN tag of this zone
 183
 184 service vlan protocol:: Allows you to choose between an 802.1q (default) or
 185   802.1ad service VLAN type.
 186
 187 mtu:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
 188   For example, you must reduce the MTU to `1496` if you physical interface MTU is
 189   `1500`.
 190
 191 [[pvesdn_zone_plugin_vxlan]]
 192 VXLAN Zones
 193 ~~~~~~~~~~~
 194
 195 The VXLAN plugin establishes a tunnel (overlay) on top of an existing
 196 network (underlay). This encapsulates layer 2 Ethernet frames within layer
 197 4 UDP datagrams, using `4789` as the default destination port. You can, for
 198 example, create a private IPv4 VXLAN network on top of public internet network
 199 nodes.
 200
 201 This is a layer 2 tunnel only, so no routing between different VNets is
 202 possible.
 203
 204 Each VNet will have a specific VXLAN ID in the range 1 - 16777215.
 205
 206 Specific EVPN configuration options:
 207
 208 peers address list:: A list of IP addresses from each node through which you
 209   want to communicate. Can also be external nodes.
 210
 211 mtu:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
 212   lower than the outgoing physical interface.
 213
 214 [[pvesdn_zone_plugin_evpn]]
 215 EVPN Zones
 216 ~~~~~~~~~~
 217
 218 This is the most complex of all the supported plugins.
 219
 220 BGP-EVPN allows you to create a routable layer 3 network. The VNet of EVPN can
 221 have an anycast IP address and/or MAC address. The bridge IP is the same on each
 222 node, meaning a virtual guest can use this address as gateway.
 223
 224 Routing can work across VNets from different zones through a VRF (Virtual
 225 Routing and Forwarding) interface.
 226
 227 The configuration options specific to EVPN are as follows:
 228
 229 VRF VXLAN tag:: This is a VXLAN-ID used for routing interconnect between VNets.
 230   It must be different than the VXLAN-ID of the VNets.
 231
 232 controller:: An EVPN-controller must to be defined first (see controller plugins
 233   section).
 234
 235 VNet MAC address:: A unique, anycast MAC address for all VNets in this zone.
 236   Will be auto-generated if not defined.
 237
 238 Exit Nodes:: Optional. This is used if you want to define some {pve} nodes as
 239   exit gateways from the EVPN network, through the real network. The configured
 240   nodes will announce a default route in the EVPN network.
 241
 242 Primary Exit Node:: Optional. If you use multiple exit nodes, this forces
 243   traffic to a primary exit node, instead of load-balancing on all nodes. This
 244   is required if you want to use SNAT or if your upstream router doesn't support
 245   ECMP.
 246
 247 Exit Nodes local routing:: Optional. This is a special option if you need to
 248   reach a VM/CT service from an exit node. (By default, the exit nodes only
 249   allow forwarding traffic between real network and EVPN network).
 250
 251 Advertise Subnets:: Optional. If you have silent VMs/CTs (for example, if you
 252   have multiple IPs and the anycast gateway doesn't see traffic from theses IPs,
 253   the IP addresses won't be able to be reach inside the EVPN network). This
 254   option will announce the full subnet in the EVPN network in this case.
 255
 256 Disable Arp-Nd Suppression:: Optional. Don't suppress ARP or ND packets.
 257   This is required if you use floating IPs in your guest VMs
 258   (IP are MAC addresses are being moved between systems).
 259
 260 Route-target import:: Optional. Allows you to import a list of external EVPN
 261   route targets. Used for cross-DC or different EVPN network interconnects.
 262
 263 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
 264   less than the maximal MTU of the outgoing physical interface.
 265
 266
 267 [[pvesdn_config_vnet]]
 268 VNets
 269 -----
 270
 271 A `VNet` is, in its basic form, a Linux bridge that will be deployed locally on
 272 the node and used for virtual machine communication.
 273
 274 The VNet configuration properties are:
 275
 276 ID:: An 8 character ID to name and identify a VNet
 277
 278 Alias:: Optional longer name, if the ID isn't enough
 279
 280 Zone:: The associated zone for this VNet
 281
 282 Tag:: The unique VLAN or VXLAN ID
 283
 284 VLAN Aware:: Enable adding an extra VLAN tag in the virtual machine or
 285 container's vNIC configuration, to allow the guest OS to manage the VLAN's tag.
 286
 287 [[pvesdn_config_subnet]]
 288 Subnets
 289 ~~~~~~~~
 290
 291 A subnetwork (subnet) allows you to define a specific IP network
 292 (IPv4 or IPv6). For each VNet, you can define one or more subnets.
 293
 294 A subnet can be used to:
 295
 296 * Restrict the IP addresses you can define on a specific VNet
 297 * Assign routes/gateways on a VNet in layer 3 zones
 298 * Enable SNAT on a VNet in layer 3 zones
 299 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
 300 * DNS registration through DNS plugins
 301
 302 If an IPAM server is associated with the subnet zone, the subnet prefix will be
 303 automatically registered in the IPAM.
 304
 305 Subnet properties are:
 306
 307 ID:: A CIDR network address, for example 10.0.0.0/8
 308
 309 Gateway:: The IP address of the network's default gateway. On layer 3 zones
 310   (Simple/EVPN plugins), it will be deployed on the VNet.
 311
 312 SNAT:: Optional. Enable SNAT for layer 3 zones (Simple/EVPN plugins), for this
 313   subnet. The subnet's source IP will be NATted to server's outgoing interface/IP.
 314   On EVPN zones, this is only done on EVPN gateway-nodes.
 315
 316 Dnszoneprefix:: Optional. Add a prefix to the domain registration, like
 317 <hostname>.prefix.<domain>
 318
 319 [[pvesdn_config_controllers]]
 320 Controllers
 321 -----------
 322
 323 Some zone types need an external controller to manage the VNet control-plane.
 324 Currently this is only required for the `bgp-evpn` zone plugin.
 325
 326 [[pvesdn_controller_plugin_evpn]]
 327 EVPN Controller
 328 ~~~~~~~~~~~~~~~
 329
 330 For `BGP-EVPN`, we need a controller to manage the control plane.
 331 The currently supported software controller is the "frr" router.
 332 You may need to install it on each node where you want to deploy EVPN zones.
 333
 334 ----
 335 apt install frr frr-pythontools
 336 ----
 337
 338 Configuration options:
 339
 340 asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
 341   number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
 342   breaking global routing by mistake.
 343
 344 peers:: An IP list of all nodes where you want to communicate for the EVPN
 345   (could also be external nodes or route reflectors servers)
 346
 347
 348 [[pvesdn_controller_plugin_BGP]]
 349 BGP Controller
 350 ~~~~~~~~~~~~~~~
 351
 352 The BGP controller is not used directly by a zone.
 353 You can use it to configure FRR to manage BGP peers.
 354
 355 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
 356 It can also be used to export evpn routes to a external bgp peer.
 357
 358 NOTE: By default, for a simple full mesh evpn, you don't need to define any extra
 359 BGP Controller.
 360
 361 Configuration options:
 362
 363 node:: The node of this BGP controller
 364
 365 asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
 366   number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
 367   you could break global routing by mistake.
 368
 369 peers:: A list of peer IP addresses you want to communicate with using the
 370   underlying BGP network.
 371
 372 ebgp:: If your peer's remote-AS is different, this enables EBGP.
 373
 374 loopback:: Use a loopback or dummy interface as the source of the EVPN network
 375   (for multipath).
 376
 377 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
 378   not directly connected or they use loopback.
 379
 380 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
 381
 382
 383 [[pvesdn_controller_plugin_ISIS]]
 384 ISIS Controller
 385 ~~~~~~~~~~~~~~~
 386
 387 The ISIS controller is not used directly by a zone.
 388 You can use it to configure FRR to export evpn routes to an ISIS domain.
 389
 390 Configuration options:
 391
 392 node:: The node of this ISIS controller.
 393
 394 domain:: A unique ISIS domain.
 395
 396 network entity title:: A Unique ISIS network address that identifies this node.
 397
 398 interfaces:: A list of physical interface(s) used by ISIS.
 399
 400 loopback:: Use a loopback or dummy interface as the source of the EVPN network
 401   (for multipath).
 402
 403 [[pvesdn_config_ipam]]
 404 IPAMs
 405 -----
 406
 407 IPAM (IP Address Management) tools are used to manage/assign the IP addresses of
 408 guests on the network. It can be used to find free IP addresses when you create
 409 a VM/CT for example (not yet implemented).
 410
 411 An IPAM can be associated with one or more zones, to provide IP addresses
 412 for all subnets defined in those zones.
 413
 414 [[pvesdn_ipam_plugin_pveipam]]
 415 {pve} IPAM Plugin
 416 ~~~~~~~~~~~~~~~~~
 417
 418 This is the default internal IPAM for your {pve} cluster, if you don't have
 419 external IPAM software.
 420
 421 [[pvesdn_ipam_plugin_phpipam]]
 422 phpIPAM Plugin
 423 ~~~~~~~~~~~~~~
 424 https://phpipam.net/
 425
 426 You need to create an application in phpIPAM and add an API token with admin
 427 privileges.
 428
 429 The phpIPAM configuration properties are:
 430
 431 url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
 432
 433 token:: An API access token
 434
 435 section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
 436   installations use `sectionid=1` for customers.
 437
 438 [[pvesdn_ipam_plugin_netbox]]
 439 NetBox IPAM Plugin
 440 ~~~~~~~~~~~~~~~~~~
 441
 442 NetBox is an IP address management (IPAM) and datacenter infrastructure
 443 management (DCIM) tool. See the source code repository for details:
 444 https://github.com/netbox-community/netbox
 445
 446 You need to create an API token in NetBox to use it:
 447 https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
 448
 449 The NetBox configuration properties are:
 450
 451 url:: The REST API endpoint: `http://yournetbox.domain.com/api`
 452
 453 token:: An API access token
 454
 455 [[pvesdn_config_dns]]
 456 DNS
 457 ---
 458
 459 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
 460 of your hostname and IP address. A DNS configuration is associated with one or
 461 more zones, to provide DNS registration for all the subnet IPs configured for
 462 a zone.
 463
 464 [[pvesdn_dns_plugin_powerdns]]
 465 PowerDNS Plugin
 466 ~~~~~~~~~~~~~~~
 467 https://doc.powerdns.com/authoritative/http-api/index.html
 468
 469 You need to enable the web server and the API in your PowerDNS config:
 470
 471 ----
 472 api=yes
 473 api-key=arandomgeneratedstring
 474 webserver=yes
 475 webserver-port=8081
 476 ----
 477
 478 The PowerDNS configuration options are:
 479
 480 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
 481
 482 key:: An API access key
 483
 484 ttl:: The default TTL for records
 485
 486
 487 Examples
 488 --------
 489
 490 [[pvesdn_setup_example_vlan]]
 491 VLAN Setup Example
 492 ~~~~~~~~~~~~~~~~~~
 493
 494 TIP: While we show plaintext configuration content here, almost everything
 495 should be configurable using the web-interface only.
 496
 497 Node1: /etc/network/interfaces
 498
 499 ----
 500 auto vmbr0
 501 iface vmbr0 inet manual
 502         bridge-ports eno1
 503         bridge-stp off
 504         bridge-fd 0
 505         bridge-vlan-aware yes
 506         bridge-vids 2-4094
 507
 508 #management ip on vlan100
 509 auto vmbr0.100
 510 iface vmbr0.100 inet static
 511         address 192.168.0.1/24
 512
 513 source /etc/network/interfaces.d/*
 514 ----
 515
 516 Node2: /etc/network/interfaces
 517
 518 ----
 519 auto vmbr0
 520 iface vmbr0 inet manual
 521         bridge-ports eno1
 522         bridge-stp off
 523         bridge-fd 0
 524         bridge-vlan-aware yes
 525         bridge-vids 2-4094
 526
 527 #management ip on vlan100
 528 auto vmbr0.100
 529 iface vmbr0.100 inet static
 530         address 192.168.0.2/24
 531
 532 source /etc/network/interfaces.d/*
 533 ----
 534
 535 Create a VLAN zone named `myvlanzone':
 536
 537 ----
 538 id: myvlanzone
 539 bridge: vmbr0
 540 ----
 541
 542 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
 543 `myvlanzone' as its zone.
 544
 545 ----
 546 id: myvnet1
 547 zone: myvlanzone
 548 tag: 10
 549 ----
 550
 551 Apply the configuration through the main SDN panel, to create VNets locally on
 552 each node.
 553
 554 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
 555
 556 Use the following network configuration for this VM:
 557
 558 ----
 559 auto eth0
 560 iface eth0 inet static
 561         address 10.0.3.100/24
 562 ----
 563
 564 Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
 565 `myvnet1' as vm1.
 566
 567 Use the following network configuration for this VM:
 568
 569 ----
 570 auto eth0
 571 iface eth0 inet static
 572         address 10.0.3.101/24
 573 ----
 574
 575 Following this, you should be able to ping between both VMs over that network.
 576
 577
 578 [[pvesdn_setup_example_qinq]]
 579 QinQ Setup Example
 580 ~~~~~~~~~~~~~~~~~~
 581
 582 TIP: While we show plaintext configuration content here, almost everything
 583 should be configurable using the web-interface only.
 584
 585 Node1: /etc/network/interfaces
 586
 587 ----
 588 auto vmbr0
 589 iface vmbr0 inet manual
 590         bridge-ports eno1
 591         bridge-stp off
 592         bridge-fd 0
 593         bridge-vlan-aware yes
 594         bridge-vids 2-4094
 595
 596 #management ip on vlan100
 597 auto vmbr0.100
 598 iface vmbr0.100 inet static
 599         address 192.168.0.1/24
 600
 601 source /etc/network/interfaces.d/*
 602 ----
 603
 604 Node2: /etc/network/interfaces
 605
 606 ----
 607 auto vmbr0
 608 iface vmbr0 inet manual
 609         bridge-ports eno1
 610         bridge-stp off
 611         bridge-fd 0
 612         bridge-vlan-aware yes
 613         bridge-vids 2-4094
 614
 615 #management ip on vlan100
 616 auto vmbr0.100
 617 iface vmbr0.100 inet static
 618         address 192.168.0.2/24
 619
 620 source /etc/network/interfaces.d/*
 621 ----
 622
 623 Create a QinQ zone named `qinqzone1' with service VLAN 20
 624
 625 ----
 626 id: qinqzone1
 627 bridge: vmbr0
 628 service vlan: 20
 629 ----
 630
 631 Create another QinQ zone named `qinqzone2' with service VLAN 30
 632
 633 ----
 634 id: qinqzone2
 635 bridge: vmbr0
 636 service vlan: 30
 637 ----
 638
 639 Create a VNet named `myvnet1' with customer VLAN-ID 100 on the previously
 640 created `qinqzone1' zone.
 641
 642 ----
 643 id: myvnet1
 644 zone: qinqzone1
 645 tag: 100
 646 ----
 647
 648 Create a `myvnet2' with customer VLAN-ID 100 on the previously created
 649 `qinqzone2' zone.
 650
 651 ----
 652 id: myvnet2
 653 zone: qinqzone2
 654 tag: 100
 655 ----
 656
 657 Apply the configuration on the main SDN web-interface panel to create VNets
 658 locally on each nodes.
 659
 660 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
 661
 662 Use the following network configuration for this VM:
 663
 664 ----
 665 auto eth0
 666 iface eth0 inet static
 667         address 10.0.3.100/24
 668 ----
 669
 670 Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
 671 `myvnet1' as vm1.
 672
 673 Use the following network configuration for this VM:
 674
 675 ----
 676 auto eth0
 677 iface eth0 inet static
 678         address 10.0.3.101/24
 679 ----
 680
 681 Create a third virtual machine (vm3) on node1, with a vNIC on the other VNet
 682 `myvnet2'.
 683
 684 Use the following network configuration for this VM:
 685
 686 ----
 687 auto eth0
 688 iface eth0 inet static
 689         address 10.0.3.102/24
 690 ----
 691
 692 Create another virtual machine (vm4) on node2, with a vNIC on the same VNet
 693 `myvnet2' as vm3.
 694
 695 Use the following network configuration for this VM:
 696
 697 ----
 698 auto eth0
 699 iface eth0 inet static
 700         address 10.0.3.103/24
 701 ----
 702
 703 Then, you should be able to ping between the VMs 'vm1' and 'vm2', as well as
 704 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
 705 'vm3' or 'vm4', as they are on a different zone with a different service-vlan.
 706
 707
 708 [[pvesdn_setup_example_vxlan]]
 709 VXLAN Setup Example
 710 ~~~~~~~~~~~~~~~~~~~
 711
 712 TIP: While we show plaintext configuration content here, almost everything
 713 is configurable through the web-interface.
 714
 715 node1: /etc/network/interfaces
 716
 717 ----
 718 auto vmbr0
 719 iface vmbr0 inet static
 720         address 192.168.0.1/24
 721         gateway 192.168.0.254
 722         bridge-ports eno1
 723         bridge-stp off
 724         bridge-fd 0
 725         mtu 1500
 726
 727 source /etc/network/interfaces.d/*
 728 ----
 729
 730 node2: /etc/network/interfaces
 731
 732 ----
 733 auto vmbr0
 734 iface vmbr0 inet static
 735         address 192.168.0.2/24
 736         gateway 192.168.0.254
 737         bridge-ports eno1
 738         bridge-stp off
 739         bridge-fd 0
 740         mtu 1500
 741
 742 source /etc/network/interfaces.d/*
 743 ----
 744
 745 node3: /etc/network/interfaces
 746
 747 ----
 748 auto vmbr0
 749 iface vmbr0 inet static
 750         address 192.168.0.3/24
 751         gateway 192.168.0.254
 752         bridge-ports eno1
 753         bridge-stp off
 754         bridge-fd 0
 755         mtu 1500
 756
 757 source /etc/network/interfaces.d/*
 758 ----
 759
 760 Create a VXLAN zone named `myvxlanzone', using a lower MTU to ensure the extra
 761 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
 762 the nodes to the peer address list.
 763
 764 ----
 765 id: myvxlanzone
 766 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
 767 mtu: 1450
 768 ----
 769
 770 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
 771 previously.
 772
 773 ----
 774 id: myvnet1
 775 zone: myvxlanzone
 776 tag: 100000
 777 ----
 778
 779 Apply the configuration on the main SDN web-interface panel to create VNets
 780 locally on each nodes.
 781
 782 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
 783
 784 Use the following network configuration for this VM (note the lower MTU).
 785
 786 ----
 787 auto eth0
 788 iface eth0 inet static
 789         address 10.0.3.100/24
 790         mtu 1450
 791 ----
 792
 793 Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet
 794 `myvnet1' as vm1.
 795
 796 Use the following network configuration for this VM:
 797
 798 ----
 799 auto eth0
 800 iface eth0 inet static
 801         address 10.0.3.101/24
 802         mtu 1450
 803 ----
 804
 805 Then, you should be able to ping between between 'vm1' and 'vm2'.
 806
 807
 808 [[pvesdn_setup_example_evpn]]
 809 EVPN Setup Example
 810 ~~~~~~~~~~~~~~~~~~
 811
 812 node1: /etc/network/interfaces
 813
 814 ----
 815 auto vmbr0
 816 iface vmbr0 inet static
 817         address 192.168.0.1/24
 818         gateway 192.168.0.254
 819         bridge-ports eno1
 820         bridge-stp off
 821         bridge-fd 0
 822         mtu 1500
 823
 824 source /etc/network/interfaces.d/*
 825 ----
 826
 827 node2: /etc/network/interfaces
 828
 829 ----
 830 auto vmbr0
 831 iface vmbr0 inet static
 832         address 192.168.0.2/24
 833         gateway 192.168.0.254
 834         bridge-ports eno1
 835         bridge-stp off
 836         bridge-fd 0
 837         mtu 1500
 838
 839 source /etc/network/interfaces.d/*
 840 ----
 841
 842 node3: /etc/network/interfaces
 843
 844 ----
 845 auto vmbr0
 846 iface vmbr0 inet static
 847         address 192.168.0.3/24
 848         gateway 192.168.0.254
 849         bridge-ports eno1
 850         bridge-stp off
 851         bridge-fd 0
 852         mtu 1500
 853
 854 source /etc/network/interfaces.d/*
 855 ----
 856
 857 Create an EVPN controller, using a private ASN number and the above node
 858 addresses as peers.
 859
 860 ----
 861 id: myevpnctl
 862 asn: 65000
 863 peers: 192.168.0.1,192.168.0.2,192.168.0.3
 864 ----
 865
 866 Create an EVPN zone named `myevpnzone', using the previously created
 867 EVPN-controller. Define 'node1' and 'node2' as exit nodes.
 868
 869 ----
 870 id: myevpnzone
 871 vrf vxlan tag: 10000
 872 controller: myevpnctl
 873 mtu: 1450
 874 vnet mac address: 32:F4:05:FE:6C:0A
 875 exitnodes: node1,node2
 876 ----
 877
 878 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
 879 ----
 880 id: myvnet1
 881 zone: myevpnzone
 882 tag: 11000
 883 ----
 884
 885 Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on `myvnet1`.
 886
 887 ----
 888 subnet: 10.0.1.0/24
 889 gateway: 10.0.1.1
 890 ----
 891
 892 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
 893 different IPv4 CIDR network.
 894
 895 ----
 896 id: myvnet2
 897 zone: myevpnzone
 898 tag: 12000
 899 ----
 900
 901 Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
 902
 903 ----
 904 subnet: 10.0.2.0/24
 905 gateway: 10.0.2.1
 906 ----
 907
 908
 909 Apply the configuration from the main SDN web-interface panel to create VNets
 910 locally on each node and generate the FRR config.
 911
 912 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
 913
 914 Use the following network configuration for this VM:
 915
 916 ----
 917 auto eth0
 918 iface eth0 inet static
 919         address 10.0.1.100/24
 920         gateway 10.0.1.1   #this is the ip of the vnet1
 921         mtu 1450
 922 ----
 923
 924 Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet
 925 `myvnet2'.
 926
 927 Use the following network configuration for this VM:
 928
 929 ----
 930 auto eth0
 931 iface eth0 inet static
 932         address 10.0.2.100/24
 933         gateway 10.0.2.1   #this is the ip of the myvnet2
 934         mtu 1450
 935 ----
 936
 937
 938 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
 939
 940 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
 941 will go to the configured 'myvnet2' gateway, then will be routed to the exit
 942 nodes ('node1' or 'node2') and from there it will leave those nodes over the
 943 default gateway configured on node1 or node2.
 944
 945 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
 946 networks to node1 and node2 on your external gateway, so that the public network
 947 can reply back.
 948
 949 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 950 and 10.0.2.0/24 in this example), will be announced dynamically.
 951
 952
 953 Notes
 954 -----
 955
 956 Multiple EVPN Exit Nodes
 957 ~~~~~~~~~~~~~~~~~~~~~~~~
 958
 959 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
 960 Reverse Path Filter) option, because packets can arrive at one node but go out
 961 from another node.
 962
 963 .sysctl.conf disabling `rp_filter`
 964 -----
 965 net.ipv4.conf.default.rp_filter=0
 966 net.ipv4.conf.all.rp_filter=0
 967 -----
 968
 969 VXLAN IPSEC Encryption
 970 ~~~~~~~~~~~~~~~~~~~~~~
 971
 972 If you need to add encryption on top of a VXLAN, it's possible to do so with
 973 IPSEC, through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
 974 or 80 bytes (IPv6) to handle encryption.
 975
 976 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
 977 + 50 (VXLAN) == 1500).
 978
 979 .Install strongswan
 980 ----
 981 apt install strongswan
 982 ----
 983
 984 Add configuration to `/etc/ipsec.conf'. We only need to encrypt traffic from
 985 the VXLAN UDP port '4789'.
 986
 987 ----
 988 conn %default
 989     ike=aes256-sha1-modp1024!  # the fastest, but reasonably secure cipher on modern HW
 990     esp=aes256-sha1!
 991     leftfirewall=yes           # this is necessary when using Proxmox VE firewall rules
 992
 993 conn output
 994     rightsubnet=%dynamic[udp/4789]
 995     right=%any
 996     type=transport
 997     authby=psk
 998     auto=route
 999
1000 conn input
1001     leftsubnet=%dynamic[udp/4789]
1002     type=transport
1003     authby=psk
1004     auto=route
1005 ----
1006
1007 Then generate a pre-shared key with:
1008
1009 ----
1010 openssl rand -base64 128
1011 ----
1012
1013 and add the key to `/etc/ipsec.secrets', so that the file contents looks like:
1014
1015 ----
1016 : PSK <generatedbase64key>
1017 ----
1018
1019 You need to copy the PSK and the configuration onto the other nodes.