pvesdn.adoc

   1 [[chapter_pvesdn]]
   2 Software-Defined Network
   3 ========================
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 The **S**oftware-**D**efined **N**etwork (SDN) feature in {pve} enables the
   9 creation of virtual zones and networks (VNets). This functionality simplifies
  10 advanced networking configurations and multitenancy setup."
  11
  12 [[pvesdn_overview]]
  13 Introduction
  14 ------------
  15
  16 The {pve} SDN allows for separation and fine-grained control of virtual guest
  17 networks, using flexible, software-controlled configurations.
  18
  19 Separation is managed through *zones*, virtual networks (*VNets*), and
  20 *subnets*.  A zone is its own virtually separated network area.  A VNet is a
  21 virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
  22
  23 Depending on the type of the zone, the network behaves differently and offers
  24 specific features, advantages, and limitations.
  25
  26 Use cases for SDN range from an isolated private network on each individual node
  27 to complex overlay networks across multiple PVE clusters on different locations.
  28
  29 After configuring an VNet in the cluster-wide datacenter SDN administration
  30 interface, it is available as a common Linux bridge, locally on each node, to be
  31 assigned to VMs and Containers.
  32
  33
  34 [[pvesdn_support_status]]
  35 Support Status
  36 --------------
  37
  38 History
  39 ~~~~~~~
  40
  41 The {pve} SDN stack has been available as an experimental feature since 2019 and
  42 has been continuously improved and tested by many developers and users.
  43 With its integration into the web interface in {pve} 6.2, a significant
  44 milestone towards broader integration was achieved.
  45 During the {pve} 7 release cycle, numerous improvements and features were added.
  46 Based on user feedback, it became apparent that the fundamental design choices
  47 and their implementation were quite sound and stable. Consequently, labeling it
  48 as `experimental' did not do justice to the state of the SDN stack.
  49 For {pve} 8, a decision was made to lay the groundwork for full integration of
  50 the SDN feature by elevating the management of networks and interfaces to a core
  51 component in the {pve} access control stack.
  52 In {pve} 8.1, two major milestones were achieved: firstly, DHCP integration was
  53 added to the IP address management (IPAM) feature, and secondly, the SDN
  54 integration is now installed by default.
  55
  56 Current Status
  57 ~~~~~~~~~~~~~~
  58
  59 The current support status for the various layers of our SDN installation is as
  60 follows:
  61
  62 - Core SDN, which includes VNet management and its integration with the {pve}
  63   stack, is fully supported.
  64 - IPAM, including DHCP management for virtual guests, is in tech preview.
  65 - Complex routing via FRRouting and controller integration are in tech preview.
  66
  67 [[pvesdn_installation]]
  68 Installation
  69 ------------
  70
  71 SDN Core
  72 ~~~~~~~~
  73
  74 Since {pve} 8.1 the core Software-Defined Network (SDN) packages are installed
  75 by default.
  76
  77 If you upgrade from an older version, you need to install the
  78 `libpve-network-perl` package on every node:
  79
  80 ----
  81 apt update
  82 apt install libpve-network-perl
  83 ----
  84
  85 NOTE: {pve} version 7.0 and above have the `ifupdown2` package installed by
  86 default. If you originally installed your system with an older version, you need
  87 to explicitly install the `ifupdown2` package.
  88 +
  89 After installation, you need to add the following line to the end of the
  90 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
  91 included and activated.
  92 +
  93 ----
  94 source /etc/network/interfaces.d/*
  95 ----
  96
  97 DHCP IPAM
  98 ~~~~~~~~~
  99
 100 The DHCP integration into the IP Address Management stack currently uses
 101 `dnsmasq` for giving out DHCP leases. This is currently opt-in.
 102
 103 To use that feature you need to install the `dnsmasq` package on every node:
 104
 105 ----
 106 apt update
 107 apt install dnsmasq
 108 ----
 109
 110 FRRouting
 111 ~~~~~~~~~
 112
 113 The {pve} SDN stack uses the https://frrouting.org/[FRRouting] project for
 114 advanced setups. This is currently opt-in.
 115
 116 To use the SDN routing integration you need to install the `frr-pythontools`
 117 package on all nodes:
 118
 119 ----
 120 apt update
 121 apt install frr-pythontools
 122 ----
 123
 124 [[pvesdn_main_configuration]]
 125 Configuration Overview
 126 ----------------------
 127
 128 Configuration is done at the web UI at datacenter level, separated into the
 129 following sections:
 130
 131 * SDN:: Here you get an overview of the current active SDN state, and you can
 132   apply all pending changes to the whole cluster.
 133
 134 * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
 135   network zones
 136
 137 * xref:pvesdn_config_vnet[VNets] VNets: Create virtual network bridges and
 138   manage subnets
 139
 140 The Options category allows adding and managing additional services to be used
 141 in your SDN setup.
 142
 143 * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
 144   in complex setups
 145
 146 * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
 147   guests
 148
 149 * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
 150   virtual guests' hostname and IP addresses
 151
 152 [[pvesdn_tech_and_config_overview]]
 153 Technology & Configuration
 154 --------------------------
 155
 156 The {pve} Software-Defined Network implementation uses standard Linux networking
 157 as much as possible. The reason for this is that modern Linux networking
 158 provides almost all needs for a feature full SDN implementation and avoids adding
 159 external dependencies and reduces the overall amount of components that can
 160 break.
 161
 162 The {pve} SDN configurations are located in `/etc/pve/sdn`, which is shared with
 163 all other cluster nodes through the {pve} xref:chapter_pmxcfs[configuration file system].
 164 Those configurations get translated to the respective configuration formats of
 165 the tools that manage the underlying network stack (for example `ifupdown2` or
 166 `frr`).
 167
 168 New changes are not immediately applied but recorded as pending first. You can
 169 then apply a set of different changes all at once in the main 'SDN' overview
 170 panel on the web interface. This system allows to roll-out various changes as
 171 single atomic one.
 172
 173 The SDN tracks the rolled-out state through the '.running-config' and '.version'
 174 files located in '/etc/pve/sdn'.
 175
 176 // TODO: extend implementation and technology details.
 177
 178 [[pvesdn_config_zone]]
 179 Zones
 180 -----
 181
 182 A zone defines a virtually separated network. Zones are restricted to
 183 specific nodes and assigned permissions, in order to restrict users to a certain
 184 zone and its contained VNets.
 185
 186 Different technologies can be used for separation:
 187
 188 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
 189
 190 * VLAN: Virtual LANs are the classic method of subdividing a LAN
 191
 192 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
 193
 194 * VXLAN: Layer 2 VXLAN network via a UDP tunnel
 195
 196 * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
 197
 198
 199 [[pvesdn_config_common_options]]
 200 Common Options
 201 ~~~~~~~~~~~~~~
 202
 203 The following options are available for all zone types:
 204
 205 Nodes:: The nodes which the zone and associated VNets should be deployed on.
 206
 207 IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
 208   zone. Optional, defaults to `pve`.
 209
 210 DNS:: DNS API server. Optional.
 211
 212 ReverseDNS:: Reverse DNS API server. Optional.
 213
 214 DNSZone:: DNS domain name. Used to register hostnames, such as
 215   `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
 216
 217
 218 [[pvesdn_zone_plugin_simple]]
 219 Simple Zones
 220 ~~~~~~~~~~~~
 221
 222 This is the simplest plugin. It will create an isolated VNet bridge.  This
 223 bridge is not linked to a physical interface, and VM traffic is only local on
 224 each the node.
 225 It can be used in NAT or routed setups.
 226
 227
 228 [[pvesdn_zone_plugin_vlan]]
 229 VLAN Zones
 230 ~~~~~~~~~~
 231
 232 The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
 233 node's physical interface.  It uses VLAN tagging defined in the VNet to isolate
 234 the network segments.  This allows connectivity of VMs between different nodes.
 235
 236 VLAN zone configuration options:
 237
 238 Bridge:: The local bridge or OVS switch, already configured on *each* node that
 239   allows node-to-node connection.
 240
 241
 242 [[pvesdn_zone_plugin_qinq]]
 243 QinQ Zones
 244 ~~~~~~~~~~
 245
 246 QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
 247 isolation.  The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
 248 whereas the inner VLAN tag is defined by the VNet.
 249
 250 NOTE: Your physical network switches must support stacked VLANs for this
 251 configuration.
 252
 253 QinQ zone configuration options:
 254
 255 Bridge:: A local, VLAN-aware bridge that is already configured on each local
 256   node
 257
 258 Service VLAN:: The main VLAN tag of this zone
 259
 260 Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
 261   802.1ad service VLAN type.
 262
 263 MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
 264   For example, you must reduce the MTU to `1496` if you physical interface MTU is
 265   `1500`.
 266
 267
 268 [[pvesdn_zone_plugin_vxlan]]
 269 VXLAN Zones
 270 ~~~~~~~~~~~
 271
 272 The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
 273 (underlay).  This encapsulates layer 2 Ethernet frames within layer 4 UDP
 274 datagrams using the default destination port `4789`.
 275
 276 You have to configure the underlay network yourself to enable UDP connectivity
 277 between all peers.
 278
 279 You can, for example, create a VXLAN overlay network on top of public internet,
 280 appearing to the VMs as if they share the same local Layer 2 network.
 281
 282 WARNING: VXLAN on its own does does not provide any encryption. When joining
 283   multiple sites via VXLAN, make sure to establish a secure connection between
 284   the site, for example by using a site-to-site VPN.
 285
 286 VXLAN zone configuration options:
 287
 288 Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
 289   can be external nodes reachable at this IP address.
 290   All nodes in the cluster need to be mentioned here.
 291
 292 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
 293   lower than the outgoing physical interface.
 294
 295
 296 [[pvesdn_zone_plugin_evpn]]
 297 EVPN Zones
 298 ~~~~~~~~~~
 299
 300 The EVPN zone creates a routable Layer 3 network, capable of spanning across
 301 multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
 302 the routing protocol.
 303
 304 The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
 305 IP is the same on each node, meaning a virtual guest can use this address as
 306 gateway.
 307
 308 Routing can work across VNets from different zones through a VRF (Virtual
 309 Routing and Forwarding) interface.
 310
 311 EVPN zone configuration options:
 312
 313 VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
 314   It must be different than the VXLAN-ID of the VNets.
 315
 316 Controller:: The EVPN-controller to use for this zone. (See controller plugins
 317   section).
 318
 319 VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
 320   zone.  Will be auto-generated if not defined.
 321
 322 Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
 323   network, through the real network.  The configured nodes will announce a
 324   default route in the EVPN network.  Optional.
 325
 326 Primary Exit Node:: If you use multiple exit nodes, force traffic through this
 327   primary exit node, instead of load-balancing on all nodes.  Optional but
 328   necessary if you want to use SNAT or if your upstream router doesn't support
 329   ECMP.
 330
 331 Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
 332   service from an exit node. (By default, the exit nodes only allow forwarding
 333   traffic between real network and EVPN network).  Optional.
 334
 335 Advertise Subnets:: Announce the full subnet in the EVPN network.
 336   If you have silent VMs/CTs (for example, if you have multiple IPs and the
 337   anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
 338   able to be reached inside the EVPN network).  Optional.
 339
 340 Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
 341   packets.  This is required if you use floating IPs in your VMs (IP and MAC
 342   addresses are being moved between systems).  Optional.
 343
 344 Route-target Import:: Allows you to import a list of external EVPN route
 345   targets. Used for cross-DC or different EVPN network interconnects.  Optional.
 346
 347 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
 348   less than the maximal MTU of the outgoing physical interface.  Optional,
 349   defaults to 1450.
 350
 351
 352 [[pvesdn_config_vnet]]
 353 VNets
 354 -----
 355
 356 After creating a virtual network (VNet) through the SDN GUI, a local network
 357 interface with the same name is available on each node. To connect a guest to the
 358 VNet, assign the interface to the guest and set the IP address accordingly.
 359
 360 Depending on the zone, these options have different meanings and are explained
 361 in the respective zone section in this document.
 362
 363 WARNING: In the current state, some options may have no effect or won't work in
 364 certain zones.
 365
 366 VNet configuration options:
 367
 368 ID:: An up to 8 character ID to identify a VNet
 369
 370 Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
 371
 372 Zone:: The associated zone for this VNet
 373
 374 Tag:: The unique VLAN or VXLAN ID
 375
 376 VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
 377   in the quest.
 378
 379
 380 [[pvesdn_config_subnet]]
 381 Subnets
 382 -------
 383
 384 A subnet define a specific IP range, described by the CIDR network address.
 385 Each VNet, can have one or more subnets.
 386
 387 A subnet can be used to:
 388
 389 * Restrict the IP addresses you can define on a specific VNet
 390 * Assign routes/gateways on a VNet in layer 3 zones
 391 * Enable SNAT on a VNet in layer 3 zones
 392 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
 393 * DNS registration through DNS plugins
 394
 395 If an IPAM server is associated with the subnet zone, the subnet prefix will be
 396 automatically registered in the IPAM.
 397
 398 Subnet configuration options:
 399
 400 ID:: A CIDR network address, for example 10.0.0.0/8
 401
 402 Gateway:: The IP address of the network's default gateway. On layer 3 zones
 403   (Simple/EVPN plugins), it will be deployed on the VNet.
 404
 405 SNAT:: Enable Source NAT which allows VMs from inside a
 406   VNet to connect to the outside network by forwarding the packets to the nodes
 407   outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
 408   Optional.
 409
 410 DNS Zone Prefix:: Add a prefix to the domain registration, like
 411   <hostname>.prefix.<domain>  Optional.
 412
 413
 414 [[pvesdn_config_controllers]]
 415 Controllers
 416 -----------
 417
 418 Some zones implement a separated control and data plane that require an external
 419 external controller to manage the VNet's control plane.
 420
 421 Currently, only the `EVPN` zone requires an external controller.
 422
 423
 424 [[pvesdn_controller_plugin_evpn]]
 425 EVPN Controller
 426 ~~~~~~~~~~~~~~~
 427
 428 The `EVPN`, zone requires an external controller to manage the control plane.
 429 The EVPN controller plugin configures the Free Range Routing (frr) router.
 430
 431 To enable the EVPN controller, you need to install frr on every node that shall
 432 participate in the EVPN zone.
 433
 434 ----
 435 apt install frr frr-pythontools
 436 ----
 437
 438 EVPN controller configuration options:
 439
 440 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
 441   number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
 442   breaking global routing by mistake.
 443
 444 Peers:: An IP list of all nodes that are part of the EVPN zone.  (could also be
 445   external nodes or route reflector servers)
 446
 447
 448 [[pvesdn_controller_plugin_BGP]]
 449 BGP Controller
 450 ~~~~~~~~~~~~~~
 451
 452 The BGP controller is not used directly by a zone.
 453 You can use it to configure FRR to manage BGP peers.
 454
 455 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
 456 It can also be used to export EVPN routes to an external BGP peer.
 457
 458 NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
 459 controller.
 460
 461 BGP controller configuration options:
 462
 463 Node:: The node of this BGP controller
 464
 465 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
 466   number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
 467   you could break global routing by mistake.
 468
 469 Peer:: A list of peer IP addresses you want to communicate with using the
 470   underlying BGP network.
 471
 472 EBGP:: If your peer's remote-AS is different, this enables EBGP.
 473
 474 Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
 475   (for multipath).
 476
 477 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
 478   not directly connected or they use loopback.
 479
 480 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
 481
 482
 483 [[pvesdn_controller_plugin_ISIS]]
 484 ISIS Controller
 485 ~~~~~~~~~~~~~~~
 486
 487 The ISIS controller is not used directly by a zone.
 488 You can use it to configure FRR to export EVPN routes to an ISIS domain.
 489
 490 ISIS controller configuration options:
 491
 492 Node:: The node of this ISIS controller.
 493
 494 Domain:: A unique ISIS domain.
 495
 496 Network Entity Title:: A Unique ISIS network address that identifies this node.
 497
 498 Interfaces:: A list of physical interface(s) used by ISIS.
 499
 500 Loopback:: Use a loopback or dummy interface as the source of the EVPN network
 501   (for multipath).
 502
 503
 504 [[pvesdn_config_ipam]]
 505 IPAM
 506 ----
 507
 508 IP Address Management (IPAM) tools manage the IP addresses of clients on the
 509 network. SDN in {pve} uses IPAM for example to find free IP addresses for new
 510 guests.
 511
 512 A single IPAM instance can be associated with one or more zones.
 513
 514
 515 [[pvesdn_ipam_plugin_pveipam]]
 516 PVE IPAM Plugin
 517 ~~~~~~~~~~~~~~~
 518
 519 The default built-in IPAM for your {pve} cluster.
 520
 521
 522 [[pvesdn_ipam_plugin_netbox]]
 523 NetBox IPAM Plugin
 524 ~~~~~~~~~~~~~
 525
 526 link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
 527 Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
 528
 529 To integrate NetBox with {pve} SDN, create an API token in NetBox as described
 530 here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
 531
 532 The NetBox configuration properties are:
 533
 534 URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
 535
 536 Token:: An API access token
 537
 538
 539 [[pvesdn_ipam_plugin_phpipam]]
 540 phpIPAM Plugin
 541 ~~~~~~~~~~~~~~
 542
 543 In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
 544 an API token with admin privileges to the application.
 545
 546 The phpIPAM configuration properties are:
 547
 548 URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
 549
 550 Token:: An API access token
 551
 552 Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
 553   installations use `sectionid=1` for customers.
 554
 555
 556 [[pvesdn_config_dns]]
 557 DNS
 558 ---
 559
 560 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
 561 of your hostname and IP address. A DNS configuration is associated with one or
 562 more zones, to provide DNS registration for all the subnet IPs configured for
 563 a zone.
 564
 565 [[pvesdn_dns_plugin_powerdns]]
 566 PowerDNS Plugin
 567 ~~~~~~~~~~~~~~~
 568 https://doc.powerdns.com/authoritative/http-api/index.html
 569
 570 You need to enable the web server and the API in your PowerDNS config:
 571
 572 ----
 573 api=yes
 574 api-key=arandomgeneratedstring
 575 webserver=yes
 576 webserver-port=8081
 577 ----
 578
 579 The PowerDNS configuration options are:
 580
 581 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
 582
 583 key:: An API access key
 584
 585 ttl:: The default TTL for records
 586
 587
 588 [[pvesdn_setup_examples]]
 589 Examples
 590 --------
 591
 592 This section presents multiple configuration examples tailored for common SDN
 593 use cases. It aims to offer tangible implementations, providing additional
 594 details to enhance comprehension of the available configuration options.
 595
 596
 597 [[pvesdn_setup_example_simple]]
 598 Simple Zone Example
 599 ~~~~~~~~~~~~~~~~~~~
 600
 601 Simple zone networks create an isolated network for quests on a single host to
 602 connect to each other.
 603
 604 TIP: connection between quests are possible if all quests reside on a same host
 605 but cannot be reached on other nodes.
 606
 607 * Create a simple zone named `simple`.
 608 * Add a VNet names `vnet1`.
 609 * Create a Subnet with a gateway and the SNAT option enabled.
 610 * This creates a network bridge `vnet1` on the node. Assign this bridge to the
 611   quests that shall join the network and configure an IP address.
 612
 613 The network interface configuration in two VMs may look like this which allows
 614 them to communicate via the 10.0.1.0/24 network.
 615
 616 ----
 617 allow-hotplug ens19
 618 iface ens19 inet static
 619         address 10.0.1.14/24
 620 ----
 621
 622 ----
 623 allow-hotplug ens19
 624 iface ens19 inet static
 625         address 10.0.1.15/24
 626 ----
 627
 628
 629 [[pvesdn_setup_example_nat]]
 630 Source NAT Example
 631 ~~~~~~~~~~~~~~~~~~
 632
 633 If you want to allow outgoing connections for quests in the simple network zone
 634 the simple zone offers a Source NAT (SNAT) option.
 635
 636 Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
 637 Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
 638
 639 ----
 640 Subnet: 172.16.0.0/24
 641 Gateway: 172.16.0.1
 642 SNAT: checked
 643 ----
 644
 645 In the quests configure the static IP address inside the subnet's IP range.
 646
 647 The node itself will join this network with the Gateway IP '172.16.0.1' and
 648 function as the NAT gateway for quests within the subnet range.
 649
 650
 651 [[pvesdn_setup_example_vlan]]
 652 VLAN Setup Example
 653 ~~~~~~~~~~~~~~~~~~
 654
 655 When VMs on different nodes need to communicate through an isolated network, the
 656 VLAN zone allows network level isolation using VLAN tags.
 657
 658 Create a VLAN zone named `myvlanzone`:
 659
 660 ----
 661 ID: myvlanzone
 662 Bridge: vmbr0
 663 ----
 664
 665 Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
 666 `myvlanzone`.
 667
 668 ----
 669 ID: myvnet1
 670 Zone: myvlanzone
 671 Tag: 10
 672 ----
 673
 674 Apply the configuration through the main SDN panel, to create VNets locally on
 675 each node.
 676
 677 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
 678
 679 Use the following network configuration for this VM:
 680
 681 ----
 682 auto eth0
 683 iface eth0 inet static
 684         address 10.0.3.100/24
 685 ----
 686
 687 Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
 688 `myvnet1` as vm1.
 689
 690 Use the following network configuration for this VM:
 691
 692 ----
 693 auto eth0
 694 iface eth0 inet static
 695         address 10.0.3.101/24
 696 ----
 697
 698 Following this, you should be able to ping between both VMs using that network.
 699
 700
 701 [[pvesdn_setup_example_qinq]]
 702 QinQ Setup Example
 703 ~~~~~~~~~~~~~~~~~~
 704
 705
 706 This example configures two QinQ zones and adds two VMs to each zone to
 707 demonstrate the additional layer of VLAN tags which allows the configuration of
 708 more isolated VLANs.
 709
 710 A typical use case for this configuration is a hosting provider that provides an
 711 isolated network to customers for VM communication but isolates the VMs from
 712 other customers.
 713
 714 Create a QinQ zone named `qinqzone1` with service VLAN 20
 715
 716 ----
 717 ID: qinqzone1
 718 Bridge: vmbr0
 719 Service VLAN: 20
 720 ----
 721
 722 Create another QinQ zone named `qinqzone2` with service VLAN 30
 723 ----
 724 ID: qinqzone2
 725 Bridge: vmbr0
 726 Service VLAN: 30
 727 ----
 728
 729 Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
 730 `qinqzone1` zone.
 731
 732 ----
 733 ID: qinqvnet1
 734 Zone: qinqzone1
 735 Tag: 100
 736 ----
 737
 738 Create a `myvnet2` with VLAN-ID 100 on the  `qinqzone2` zone.
 739
 740 ----
 741 ID: qinqvnet2
 742 Zone: qinqzone2
 743 Tag: 100
 744 ----
 745
 746 Apply the configuration on the main SDN web-interface panel to create VNets
 747 locally on each node.
 748
 749 Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
 750 interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
 751 `qinqvnet2`.
 752
 753 Inside the VM, configure the IP addresses of the interfaces, for example via
 754 `/etc/network/interfaces`:
 755
 756 ----
 757 auto eth0
 758 iface eth0 inet static
 759         address 10.0.3.101/24
 760 ----
 761 // TODO: systemd-network example
 762 Configure all four VMs to have IP addresses from the '10.0.3.101' to
 763 '10.0.3.104' range.
 764
 765 Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
 766 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
 767 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
 768
 769
 770 [[pvesdn_setup_example_vxlan]]
 771 VXLAN Setup Example
 772 ~~~~~~~~~~~~~~~~~~~
 773
 774 The example assumes a cluster with three nodes, with the node IP addresses
 775 192.168.0.1, 192.168.0.2 and 192.168.0.3.
 776
 777 Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
 778 peer address list. Use the default MTU of 1450 or configure accordingly.
 779
 780 ----
 781 ID: myvxlanzone
 782 Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
 783 ----
 784
 785 Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
 786 previously.
 787
 788 ----
 789 ID: vxvnet1
 790 Zone: myvxlanzone
 791 Tag: 100000
 792 ----
 793
 794 Apply the configuration on the main SDN web-interface panel to create VNets
 795 locally on each nodes.
 796
 797 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
 798
 799 Use the following network configuration for this VM (note the lower MTU).
 800
 801 ----
 802 auto eth0
 803 iface eth0 inet static
 804         address 10.0.3.100/24
 805         mtu 1450
 806 ----
 807
 808 Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
 809 `vxvnet1` as vm1.
 810
 811 Use the following network configuration for this VM:
 812
 813 ----
 814 auto eth0
 815 iface eth0 inet static
 816         address 10.0.3.101/24
 817         mtu 1450
 818 ----
 819
 820 Then, you should be able to ping between between 'vm1' and 'vm2'.
 821
 822
 823 [[pvesdn_setup_example_evpn]]
 824 EVPN Setup Example
 825 ~~~~~~~~~~~~~~~~~~
 826
 827 The example assumes a cluster with three nodes (node1, node2, node3) with IP
 828 addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
 829
 830 Create an EVPN controller, using a private ASN number and the above node
 831 addresses as peers.
 832
 833 ----
 834 ID: myevpnctl
 835 ASN#: 65000
 836 Peers: 192.168.0.1,192.168.0.2,192.168.0.3
 837 ----
 838
 839 Create an EVPN zone named `myevpnzone`, assign the previously created
 840 EVPN-controller and define 'node1' and 'node2' as exit nodes.
 841
 842 ----
 843 ID: myevpnzone
 844 VRF VXLAN Tag: 10000
 845 Controller: myevpnctl
 846 MTU: 1450
 847 VNet MAC Address: 32:F4:05:FE:6C:0A
 848 Exit Nodes: node1,node2
 849 ----
 850
 851 Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
 852
 853 ----
 854 ID: myvnet1
 855 Zone: myevpnzone
 856 Tag: 11000
 857 ----
 858
 859 Create a subnet on `myvnet1`:
 860
 861 ----
 862 Subnet: 10.0.1.0/24
 863 Gateway: 10.0.1.1
 864 ----
 865
 866 Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
 867
 868 ----
 869 ID: myvnet2
 870 Zone: myevpnzone
 871 Tag: 12000
 872 ----
 873
 874 Create a different subnet on `myvnet2``:
 875
 876 ----
 877 Subnet: 10.0.2.0/24
 878 Gateway: 10.0.2.1
 879 ----
 880
 881 Apply the configuration from the main SDN web-interface panel to create VNets
 882 locally on each node and generate the FRR configuration.
 883
 884 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
 885
 886 Use the following network configuration for 'vm1':
 887
 888 ----
 889 auto eth0
 890 iface eth0 inet static
 891         address 10.0.1.100/24
 892         gateway 10.0.1.1
 893         mtu 1450
 894 ----
 895
 896 Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
 897 `myvnet2`.
 898
 899 Use the following network configuration for 'vm2':
 900
 901 ----
 902 auto eth0
 903 iface eth0 inet static
 904         address 10.0.2.100/24
 905         gateway 10.0.2.1
 906         mtu 1450
 907 ----
 908
 909
 910 Now you should be able to ping vm2 from vm1, and vm1 from vm2.
 911
 912 If you ping an external IP from 'vm2' on the non-gateway node3, the packet
 913 will go to the configured 'myvnet2' gateway, then will be routed to the exit
 914 nodes ('node1' or 'node2') and from there it will leave those nodes over the
 915 default gateway configured on node1 or node2.
 916
 917 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
 918 networks to node1 and node2 on your external gateway, so that the public network
 919 can reply back.
 920
 921 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 922 and 10.0.2.0/24 in this example), will be announced dynamically.
 923
 924
 925 [[pvesdn_notes]]
 926 Notes
 927 -----
 928
 929 Multiple EVPN Exit Nodes
 930 ~~~~~~~~~~~~~~~~~~~~~~~~
 931
 932 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
 933 Reverse Path Filter) option, because packets can arrive at one node but go out
 934 from another node.
 935
 936 Add the following to `/etc/sysctl.conf`:
 937
 938 -----
 939 net.ipv4.conf.default.rp_filter=0
 940 net.ipv4.conf.all.rp_filter=0
 941 -----
 942
 943 VXLAN IPSEC Encryption
 944 ~~~~~~~~~~~~~~~~~~~~~~
 945
 946 To add IPSEC encryption on top of a VXLAN, this example shows how to use
 947 `strongswan`.
 948
 949 You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
 950 IPv6 to handle encryption.
 951
 952 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
 953 + 50 (VXLAN) == 1500).
 954
 955 Install strongswan on the host.
 956
 957 ----
 958 apt install strongswan
 959 ----
 960
 961 Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
 962 the VXLAN UDP port '4789'.
 963
 964 ----
 965 conn %default
 966     ike=aes256-sha1-modp1024!  # the fastest, but reasonably secure cipher on modern HW
 967     esp=aes256-sha1!
 968     leftfirewall=yes           # this is necessary when using Proxmox VE firewall rules
 969
 970 conn output
 971     rightsubnet=%dynamic[udp/4789]
 972     right=%any
 973     type=transport
 974     authby=psk
 975     auto=route
 976
 977 conn input
 978     leftsubnet=%dynamic[udp/4789]
 979     type=transport
 980     authby=psk
 981     auto=route
 982 ----
 983
 984 Generate a pre-shared key with:
 985
 986 ----
 987 openssl rand -base64 128
 988 ----
 989
 990 and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
 991
 992 ----
 993 : PSK <generatedbase64key>
 994 ----
 995
 996 Copy the PSK and the configuration to all nodes participating in the VXLAN network.