[[chapter_pvesdn]]
-Software Defined Network
+Software-Defined Network
========================
ifndef::manvolnum[]
:pve-toplevel:
endif::manvolnum[]
-The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
-virtual networks (vnets) at datacenter level.
+The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
+virtual networks (VNets) at the datacenter level.
WARNING: SDN is currently an **experimental feature** in {pve}. This
-Documentation for it is also still under development, ask on our
+documentation for it is also still under development. Ask on our
xref:getting_help[mailing lists or in the forum] for questions and feedback.
Installation
------------
-To enable the experimental SDN integration, you need to install
-"libpve-network-perl" package
+To enable the experimental Software-Defined Network (SDN) integration, you need
+to install the `libpve-network-perl` package on every node:
----
+apt update
apt install libpve-network-perl
----
-You need to have `ifupdown2` package installed on each node to manage local
-configuration reloading without reboot:
+NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
+default. If you originally installed your system with an older version, you need
+to explicitly install the `ifupdown2` package.
+
+After installation, you need to add the following line to the end of the
+`/etc/network/interfaces` configuration file, so that the SDN configuration gets
+included and activated.
----
-apt install ifupdown2
+source /etc/network/interfaces.d/*
----
-Basic Overview
---------------
-The {pve} SDN allows separation and fine grained control of Virtual Guests
-networks, using flexible software controlled configurations.
+[[pvesdn_overview]]
+Overview
+--------
-Separation consists of zones, a zone is it's own virtual separated area.
-A Zone can be used by one or more 'VNets'. A 'VNet' is virtual network in a
-zone. Normally it shows up as a common Linux bridge with either a VLAN or
-'VXLAN' tag, or using layer 3 routing for control.
-The 'VNets' are deployed locally on each node, after configuration was commited
-from the cluster wide datacenter level.
+The {pve} SDN allows for separation and fine-grained control of virtual guest
+networks, using flexible, software-controlled configurations.
+Separation is managed through *zones*, virtual networks (*VNets*), and
+*subnets*. A zone is its own virtually separated network area. A VNet is a
+virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
-Main configuration
-------------------
+Depending on the type of the zone, the network behaves differently and offers
+specific features, advantages, and limitations.
-The configuration is done at datacenter (cluster-wide) level, it will be saved
-in configuration files located in the shared configuration file system:
-`/etc/pve/sdn`
+Use cases for SDN range from an isolated private network on each individual node
+to complex overlay networks across multiple PVE clusters on different locations.
-On the web-interface SDN feature have 4 main sections for the configuration
+After configuring an VNet in the cluster-wide datacenter SDN administration
+interface, it is available as a common Linux bridge, locally on each node, to be
+assigned to VMs and Containers.
-* SDN: a overview of the SDN state
-* Zones: Create and manage the virtual separated network Zones
+[[pvesdn_main_configuration]]
+Main Configuration
+~~~~~~~~~~~~~~~~~~
-* VNets: The per-node building block to provide a Zone for VMs
+Configuration is done at the web UI at datacenter level and is saved in files
+located in the shared configuration file system at `/etc/pve/sdn`.
-* Controller:
+On the web interface, SDN features the following sections:
+* SDN:: Here you get an overview of the current active SDN state, and you can
+ apply all pending changes to the whole cluster.
-[[pvesdn_config_main_sdn]]
-SDN
-~~~
+* xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
+ network zones
-This is the main status panel. Here you can see deployment status of zones on
-different nodes.
+* xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and
+ manage subnets
-There is an 'Apply' button, to push and reload local configuration on all
-cluster nodes nodes.
+The Options category allows adding and managing additional services to be used
+in your SDN setup.
+* xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
+ in complex setups
-[[pvesdn_config_zone]]
-Zones
-~~~~~
+* xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
+ guests
-A zone will define a virtually separated network.
+* xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
+ virtual guests' hostname and IP
+ addresses
-It can use different technologies for separation:
+[[pvesdn_config_main_sdn]]
+SDN
+~~~
-* VLAN: Virtual LANs are the classic method to sub-divide a LAN
+This is the main status panel. Here you can see the deployment status of zones
+on different nodes.
-* QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
+Pressing the 'Apply' button reloads the local configuration on all cluster
+nodes.
-* VXLAN: (layer2 vxlan)
+[[pvesdn_config_zone]]
+Zones
+-----
-* bgp-evpn: vxlan using layer3 border gateway protocol routing
+A zone defines a virtually separated network. Zones are restricted to
+specific nodes and assigned permissions, in order to restrict users to a certain
+zone and its contained VNets.
-You can restrict a zone to specific nodes.
+Different technologies can be used for separation:
-It's also possible to add permissions on a zone, to restrict user to use only a
-specific zone and only the VNets in that zone
+* Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
-[[pvesdn_config_vnet]]
-VNets
-~~~~~
+* VLAN: Virtual LANs are the classic method of subdividing a LAN
-A `VNet` is in its basic form just a Linux bridge that will be deployed locally
-on the node and used for Virtual Machine communication.
+* QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
-VNet properties are:
+* VXLAN: Layer 2 VXLAN network via a UDP tunnel
-* ID: a 8 characters ID to name and identify a VNet
+* EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
-* Alias: Optional longer name, if the ID isn't enough
-* Zone: The associated zone for this VNet
+[[pvesdn_config_common_options]]
+Common Options
+~~~~~~~~~~~~~~
-* Tag: The unique VLAN or VXLAN id
+The following options are available for all zone types:
-* IPv4: an anycast IPv4 address, it will be configured on the underlying bridge
- on each node part of the Zone. It's only useful for `bgp-evpn` routing.
+Nodes:: The nodes which the zone and associated VNets should be deployed on.
-* IPv6: an anycast IPv6 address, it will be configured on the underlying bridge
- on each node part of the Zone. It's only useful for `bgp-evpn` routing.
+IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
+ zone. Optional, defaults to `pve`.
+DNS:: DNS API server. Optional.
-[[pvesdn_config_controllers]]
-Controllers
-~~~~~~~~~~~
+ReverseDNS:: Reverse DNS API server. Optional.
-Some zone types need an external controller to manage the VNet control-plane.
-Currently this is only required for the `bgp-evpn` zone plugin.
+DNSZone:: DNS domain name. Used to register hostnames, such as
+ `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
-[[pvesdn_zone_plugins]]
-Zones Plugins
--------------
+[[pvesdn_zone_plugin_simple]]
+Simple Zones
+~~~~~~~~~~~~
-Common options
-~~~~~~~~~~~~~~
+This is the simplest plugin. It will create an isolated VNet bridge. This
+bridge is not linked to a physical interface, and VM traffic is only local on
+each the node.
+It can be used in NAT or routed setups.
-nodes:: Deploy and allow to use a VNets configured for this Zone only on these
-nodes.
[[pvesdn_zone_plugin_vlan]]
VLAN Zones
~~~~~~~~~~
-This is the simplest plugin, it will reuse an existing local Linux or OVS
-bridge, and manage VLANs on it.
-The benefit of using SDN module, is that you can create different zones with
-specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
+The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
+node's physical interface. It uses VLAN tagging defined in the VNet to isolate
+the network segments. This allows connectivity of VMs between different nodes.
+
+VLAN zone configuration options:
-Specific `VLAN` configuration options:
+Bridge:: The local bridge or OVS switch, already configured on *each* node that
+ allows node-to-node connection.
-bridge:: Reuse this local VLAN-aware bridge, or OVS interface, already
-configured on *each* local node.
[[pvesdn_zone_plugin_qinq]]
QinQ Zones
~~~~~~~~~~
-QinQ is stacked VLAN. The first VLAN tag defined for the zone
-(so called 'service-vlan'), and the second VLAN tag defined for the vnets
+QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
+isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
+whereas the inner VLAN tag is defined by the VNet.
-NOTE: Your physical network switchs must support stacked VLANs!
+NOTE: Your physical network switches must support stacked VLANs for this
+configuration.
-Specific QinQ configuration options:
+QinQ zone configuration options:
-bridge:: A local VLAN-aware bridge already configured on each local node
+Bridge:: A local, VLAN-aware bridge that is already configured on each local
+ node
-service vlan:: The main VLAN tag of this zone
+Service VLAN:: The main VLAN tag of this zone
+
+Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
+ 802.1ad service VLAN type.
+
+MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
+ For example, you must reduce the MTU to `1496` if you physical interface MTU is
+ `1500`.
-mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
-For example, you reduce the MTU to `1496` if you physical interface MTU is
-`1500`.
[[pvesdn_zone_plugin_vxlan]]
VXLAN Zones
~~~~~~~~~~~
-The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
-network (named underlay). It encapsulate layer 2 Ethernet frames within layer
-4 UDP datagrams, using `4789` as the default destination port. You can, for
-example, create a private IPv4 VXLAN network on top of public internet network
-nodes.
-This is a layer2 tunnel only, no routing between different VNets is possible.
+The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
+(underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
+datagrams using the default destination port `4789`.
+
+You have to configure the underlay network yourself to enable UDP connectivity
+between all peers.
+
+You can, for example, create a VXLAN overlay network on top of public internet,
+appearing to the VMs as if they share the same local Layer 2 network.
+
+WARNING: VXLAN on its own does does not provide any encryption. When joining
+ multiple sites via VXLAN, make sure to establish a secure connection between
+ the site, for example by using a site-to-site VPN.
-Each VNet will have use specific VXLAN id from the range (1 - 16777215).
+VXLAN zone configuration options:
-Specific EVPN configuration options:
+Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
+ can be external nodes reachable at this IP address.
+ All nodes in the cluster need to be mentioned here.
-peers address list:: A list of IPs from all nodes through which you want to
-communicate. Can also be external nodes.
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ lower than the outgoing physical interface.
-mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
-lower than the outgoing physical interface.
[[pvesdn_zone_plugin_evpn]]
EVPN Zones
~~~~~~~~~~
-This is the most complex of all supported plugins.
+The EVPN zone creates a routable Layer 3 network, capable of spanning across
+multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
+the routing protocol.
-BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
-have an anycast IP-address and or MAC-address. The bridge IP is the same on each
-node, with this a virtual guest can use that address as gateway.
+The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
+IP is the same on each node, meaning a virtual guest can use this address as
+gateway.
Routing can work across VNets from different zones through a VRF (Virtual
Routing and Forwarding) interface.
-Specific EVPN configuration options:
+EVPN zone configuration options:
+
+VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
+ It must be different than the VXLAN-ID of the VNets.
+
+Controller:: The EVPN-controller to use for this zone. (See controller plugins
+ section).
+
+VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
+ zone. Will be auto-generated if not defined.
+
+Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
+ network, through the real network. The configured nodes will announce a
+ default route in the EVPN network. Optional.
+
+Primary Exit Node:: If you use multiple exit nodes, force traffic through this
+ primary exit node, instead of load-balancing on all nodes. Optional but
+ necessary if you want to use SNAT or if your upstream router doesn't support
+ ECMP.
+
+Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
+ service from an exit node. (By default, the exit nodes only allow forwarding
+ traffic between real network and EVPN network). Optional.
+
+Advertise Subnets:: Announce the full subnet in the EVPN network.
+ If you have silent VMs/CTs (for example, if you have multiple IPs and the
+ anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
+ able to be reached inside the EVPN network). Optional.
+
+Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
+ packets. This is required if you use floating IPs in your VMs (IP and MAC
+ addresses are being moved between systems). Optional.
+
+Route-target Import:: Allows you to import a list of external EVPN route
+ targets. Used for cross-DC or different EVPN network interconnects. Optional.
+
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ less than the maximal MTU of the outgoing physical interface. Optional,
+ defaults to 1450.
+
+
+[[pvesdn_config_vnets]]
+VNets
+-----
+
+After creating a virtual network (VNet) through the SDN GUI, a local network
+interface with the same name is available on each node. To connect a guest to the
+VNet, assign the interface to the guest and set the IP address accordingly.
+
+Depending on the zone, these options have different meanings and are explained
+in the respective zone section in this document.
+
+WARNING: In the current state, some options may have no effect or won't work in
+certain zones.
+
+VNet configuration options:
+
+ID:: An up to 8 character ID to identify a VNet
+
+Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
+
+Zone:: The associated zone for this VNet
+
+Tag:: The unique VLAN or VXLAN ID
+
+VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
+ in the quest.
+
+
+[[pvesdn_config_subnet]]
+Subnets
+-------
+
+A subnet define a specific IP range, described by the CIDR network address.
+Each VNet, can have one or more subnets.
+
+A subnet can be used to:
-VRF VXLAN Tag:: This is a vxlan-id used for routing interconnect between vnets,
-it must be different than VXLAN-id of VNets
+* Restrict the IP addresses you can define on a specific VNet
+* Assign routes/gateways on a VNet in layer 3 zones
+* Enable SNAT on a VNet in layer 3 zones
+* Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
+* DNS registration through DNS plugins
-controller:: an EVPN-controller need to be defined first (see controller
-plugins section)
+If an IPAM server is associated with the subnet zone, the subnet prefix will be
+automatically registered in the IPAM.
-mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
-lower than the outgoing physical interface.
+Subnet configuration options:
+ID:: A CIDR network address, for example 10.0.0.0/8
+
+Gateway:: The IP address of the network's default gateway. On layer 3 zones
+ (Simple/EVPN plugins), it will be deployed on the VNet.
+
+SNAT:: Enable Source NAT which allows VMs from inside a
+ VNet to connect to the outside network by forwarding the packets to the nodes
+ outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
+ Optional.
+
+DNS Zone Prefix:: Add a prefix to the domain registration, like
+ <hostname>.prefix.<domain> Optional.
+
+
+[[pvesdn_config_controllers]]
+Controllers
+-----------
+
+Some zones implement a separated control and data plane that require an external
+external controller to manage the VNet's control plane.
+
+Currently, only the `EVPN` zone requires an external controller.
-[[pvesdn_controller_plugins]]
-Controllers Plugins
--------------------
[[pvesdn_controller_plugin_evpn]]
EVPN Controller
~~~~~~~~~~~~~~~
-For `BGP-EVPN`, we need a controller to manage the control plane.
-The currently supported software controller is the "frr" router.
-You may need to install it on each node where you want to deploy EVPN zones.
+The `EVPN`, zone requires an external controller to manage the control plane.
+The EVPN controller plugin configures the Free Range Routing (frr) router.
+
+To enable the EVPN controller, you need to install frr on every node that shall
+participate in the EVPN zone.
----
-apt install frr
+apt install frr frr-pythontools
----
-Configuration options:
+EVPN controller configuration options:
-asn:: A unique BGP ASN number. It's highly recommended to use private ASN
-number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
-breaking, or get broken, by global routing by mistake.
+ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
+ breaking global routing by mistake.
-peers:: An ip list of all nodes where you want to communicate (could be also
-external nodes or route reflectors servers)
+Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
+ external nodes or route reflector servers)
-Additionally, if you want to route traffic from a SDN BGP-EVPN network to
-external world:
-gateway-nodes:: The proxmox nodes from where the bgp-evpn traffic will exit to
-external through the nodes default gateway
+[[pvesdn_controller_plugin_BGP]]
+BGP Controller
+~~~~~~~~~~~~~~
-gateway-external-peers:: If you want that gateway nodes don't use the default
-gateway, but, for example, sent traffic to external BGP routers, which handle
-(reverse) routing then dynamically you can use. For example
-`192.168.0.253,192.168.0.254'
+The BGP controller is not used directly by a zone.
+You can use it to configure FRR to manage BGP peers.
+For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
+It can also be used to export EVPN routes to an external BGP peer.
-[[pvesdn_local_deployment_monitoring]]
-Local Deployment Monitoring
----------------------------
+NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
+controller.
-After applying the configuration through the main SDN web-interface panel,
-the local network configuration is generated locally on each node in
-`/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
+BGP controller configuration options:
-You can monitor the status of local zones and vnets through the main tree.
+Node:: The node of this BGP controller
+ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
+ you could break global routing by mistake.
-[[pvesdn_setup_example_vlan]]
-VLAN Setup Example
-------------------
+Peer:: A list of peer IP addresses you want to communicate with using the
+ underlying BGP network.
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
+EBGP:: If your peer's remote-AS is different, this enables EBGP.
-Node1: /etc/network/interfaces
+Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
+ not directly connected or they use loopback.
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.1/24
+bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
-source /etc/network/interfaces.d/*
-----
-Node2: /etc/network/interfaces
+[[pvesdn_controller_plugin_ISIS]]
+ISIS Controller
+~~~~~~~~~~~~~~~
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+The ISIS controller is not used directly by a zone.
+You can use it to configure FRR to export EVPN routes to an ISIS domain.
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.2/24
+ISIS controller configuration options:
-source /etc/network/interfaces.d/*
-----
+Node:: The node of this ISIS controller.
-Create a VLAN zone named `myvlanzone':
+Domain:: A unique ISIS domain.
-----
-id: myvlanzone
-bridge: vmbr0
-----
+Network Entity Title:: A Unique ISIS network address that identifies this node.
-Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
-`myvlanzone' as it's zone.
+Interfaces:: A list of physical interface(s) used by ISIS.
-----
-id: myvnet1
-zone: myvlanzone
-tag: 10
+Loopback:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
+
+
+[[pvesdn_config_ipam]]
+IPAM
----
-Apply the configuration through the main SDN panel, to create VNets locally on
-each nodes.
+IP Address Management (IPAM) tools manage the IP addresses of clients on the
+network. SDN in {pve} uses IPAM for example to find free IP addresses for new
+guests.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+A single IPAM instance can be associated with one or more zones.
-Use the following network configuration for this VM:
-----
-auto eth0
-iface eth0 inet static
- address 10.0.3.100/24
-----
+[[pvesdn_ipam_plugin_pveipam]]
+PVE IPAM Plugin
+~~~~~~~~~~~~~~~
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
-`myvnet1' as vm1.
+The default built-in IPAM for your {pve} cluster.
-Use the following network configuration for this VM:
-----
-auto eth0
-iface eth0 inet static
- address 10.0.3.101/24
-----
+[[pvesdn_ipam_plugin_netbox]]
+NetBox IPAM Plugin
+~~~~~~~~~~~~~
-Then, you should be able to ping between both VMs over that network.
+link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
+Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
+To integrate NetBox with {pve} SDN, create an API token in NetBox as described
+here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
-[[pvesdn_setup_example_qinq]]
-QinQ Setup Example
-------------------
+The NetBox configuration properties are:
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
+URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
-Node1: /etc/network/interfaces
+Token:: An API access token
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.1/24
+[[pvesdn_ipam_plugin_phpipam]]
+phpIPAM Plugin
+~~~~~~~~~~~~~~
-source /etc/network/interfaces.d/*
-----
+In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
+an API token with admin privileges to the application.
-Node2: /etc/network/interfaces
+The phpIPAM configuration properties are:
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.2/24
+Token:: An API access token
-source /etc/network/interfaces.d/*
-----
+Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
+ installations use `sectionid=1` for customers.
+
+
+[[pvesdn_config_dns]]
+DNS
+---
+
+The DNS plugin in {pve} SDN is used to define a DNS API server for registration
+of your hostname and IP address. A DNS configuration is associated with one or
+more zones, to provide DNS registration for all the subnet IPs configured for
+a zone.
+
+[[pvesdn_dns_plugin_powerdns]]
+PowerDNS Plugin
+~~~~~~~~~~~~~~~
+https://doc.powerdns.com/authoritative/http-api/index.html
-Create an QinQ zone named `qinqzone1' with service VLAN 20
+You need to enable the web server and the API in your PowerDNS config:
----
-id: qinqzone1
-bridge: vmbr0
-service vlan: 20
+api=yes
+api-key=arandomgeneratedstring
+webserver=yes
+webserver-port=8081
----
-Create another QinQ zone named `qinqzone2' with service VLAN 30
+The PowerDNS configuration options are:
+
+url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
+
+key:: An API access key
+
+ttl:: The default TTL for records
+
+
+[[pvesdn_setup_examples]]
+Examples
+--------
+
+This section presents multiple configuration examples tailored for common SDN
+use cases. It aims to offer tangible implementations, providing additional
+details to enhance comprehension of the available configuration options.
+
+
+[[pvesdn_setup_example_simple]]
+Simple Zone Example
+~~~~~~~~~~~~~~~~~~~
+
+Simple zone networks create an isolated network for quests on a single host to
+connect to each other.
+
+TIP: connection between quests are possible if all quests reside on a same host
+but cannot be reached on other nodes.
+
+* Create a simple zone named `simple`.
+* Add a VNet names `vnet1`.
+* Create a Subnet with a gateway and the SNAT option enabled.
+* This creates a network bridge `vnet1` on the node. Assign this bridge to the
+ quests that shall join the network and configure an IP address.
+
+The network interface configuration in two VMs may look like this which allows
+them to communicate via the 10.0.1.0/24 network.
----
-id: qinqzone2
-bridge: vmbr0
-service vlan: 30
+allow-hotplug ens19
+iface ens19 inet static
+ address 10.0.1.14/24
----
-Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
-created `qinqzone1' zone.
-
----
-id: myvnet1
-zone: qinqzone1
-tag: 100
+allow-hotplug ens19
+iface ens19 inet static
+ address 10.0.1.15/24
----
-Create a `myvnet2' with customer VLAN-id 100 on the previously created
-`qinqzone2' zone.
+
+[[pvesdn_setup_example_nat]]
+Source NAT Example
+~~~~~~~~~~~~~~~~~~
+
+If you want to allow outgoing connections for quests in the simple network zone
+the simple zone offers a Source NAT (SNAT) option.
+
+Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
+Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
----
-id: myvnet2
-zone: qinqzone1
-tag: 100
+Subnet: 172.16.0.0/24
+Gateway: 172.16.0.1
+SNAT: checked
----
-Apply the configuration on the main SDN web-interface panel to create VNets
-locally on each nodes.
+In the quests configure the static IP address inside the subnet's IP range.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+The node itself will join this network with the Gateway IP '172.16.0.1' and
+function as the NAT gateway for quests within the subnet range.
-Use the following network configuration for this VM:
+
+[[pvesdn_setup_example_vlan]]
+VLAN Setup Example
+~~~~~~~~~~~~~~~~~~
+
+When VMs on different nodes need to communicate through an isolated network, the
+VLAN zone allows network level isolation using VLAN tags.
+
+Create a VLAN zone named `myvlanzone`:
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.100/24
+ID: myvlanzone
+Bridge: vmbr0
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
-`myvnet1' as vm1.
-
-Use the following network configuration for this VM:
+Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
+`myvlanzone`.
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.101/24
+ID: myvnet1
+Zone: myvlanzone
+Tag: 10
----
-Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
-`myvnet2'.
+Apply the configuration through the main SDN panel, to create VNets locally on
+each node.
+
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
Use the following network configuration for this VM:
----
auto eth0
iface eth0 inet static
- address 10.0.3.102/24
+ address 10.0.3.100/24
----
-Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
-`myvnet2' as vm3.
+Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
+`myvnet1` as vm1.
Use the following network configuration for this VM:
----
auto eth0
iface eth0 inet static
- address 10.0.3.103/24
+ address 10.0.3.101/24
----
-Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
-between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
-or 'vm4', as they are on a different zone with different service-vlan.
+Following this, you should be able to ping between both VMs using that network.
-[[pvesdn_setup_example_vxlan]]
-VXLAN Setup Example
--------------------
+[[pvesdn_setup_example_qinq]]
+QinQ Setup Example
+~~~~~~~~~~~~~~~~~~
+
+
+This example configures two QinQ zones and adds two VMs to each zone to
+demonstrate the additional layer of VLAN tags which allows the configuration of
+more isolated VLANs.
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
+A typical use case for this configuration is a hosting provider that provides an
+isolated network to customers for VM communication but isolates the VMs from
+other customers.
-node1: /etc/network/interfaces
+Create a QinQ zone named `qinqzone1` with service VLAN 20
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.1/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+ID: qinqzone1
+Bridge: vmbr0
+Service VLAN: 20
+----
-source /etc/network/interfaces.d/*
+Create another QinQ zone named `qinqzone2` with service VLAN 30
+----
+ID: qinqzone2
+Bridge: vmbr0
+Service VLAN: 30
----
-node2: /etc/network/interfaces
+Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
+`qinqzone1` zone.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.2/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: qinqvnet1
+Zone: qinqzone1
+Tag: 100
----
-node3: /etc/network/interfaces
+Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.3/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+ID: qinqvnet2
+Zone: qinqzone2
+Tag: 100
+----
+
+Apply the configuration on the main SDN web-interface panel to create VNets
+locally on each node.
+
+Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
+interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
+`qinqvnet2`.
+
+Inside the VM, configure the IP addresses of the interfaces, for example via
+`/etc/network/interfaces`:
-source /etc/network/interfaces.d/*
----
+auto eth0
+iface eth0 inet static
+ address 10.0.3.101/24
+----
+// TODO: systemd-network example
+Configure all four VMs to have IP addresses from the '10.0.3.101' to
+'10.0.3.104' range.
+
+Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
+between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
+'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
+
+
+[[pvesdn_setup_example_vxlan]]
+VXLAN Setup Example
+~~~~~~~~~~~~~~~~~~~
-Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
-50 bytes of the VXLAN header can fit. Add all previously configured IPs from
-the nodes as peer address list.
+The example assumes a cluster with three nodes, with the node IP addresses
+192.168.0.1, 192.168.0.2 and 192.168.0.3.
+
+Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
+peer address list. Use the default MTU of 1450 or configure accordingly.
----
-id: myvxlanzone
-peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
-mtu: 1450
+ID: myvxlanzone
+Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
----
-Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
+Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
previously.
----
-id: myvnet1
-zone: myvxlanzone
-tag: 100000
+ID: vxvnet1
+Zone: myvxlanzone
+Tag: 100000
----
Apply the configuration on the main SDN web-interface panel to create VNets
locally on each nodes.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
-Use the following network configuration for this VM, note the lower MTU here.
+Use the following network configuration for this VM (note the lower MTU).
----
auto eth0
iface eth0 inet static
- address 10.0.3.100/24
- mtu 1450
+ address 10.0.3.100/24
+ mtu 1450
----
-Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
-`myvnet1' as vm1.
+Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
+`vxvnet1` as vm1.
Use the following network configuration for this VM:
----
auto eth0
iface eth0 inet static
- address 10.0.3.101/24
- mtu 1450
+ address 10.0.3.101/24
+ mtu 1450
----
Then, you should be able to ping between between 'vm1' and 'vm2'.
[[pvesdn_setup_example_evpn]]
EVPN Setup Example
-------------------
-
-node1: /etc/network/interfaces
-
-----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.1/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+~~~~~~~~~~~~~~~~~~
-source /etc/network/interfaces.d/*
-----
+The example assumes a cluster with three nodes (node1, node2, node3) with IP
+addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
-node2: /etc/network/interfaces
+Create an EVPN controller, using a private ASN number and the above node
+addresses as peers.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.2/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: myevpnctl
+ASN#: 65000
+Peers: 192.168.0.1,192.168.0.2,192.168.0.3
----
-node3: /etc/network/interfaces
+Create an EVPN zone named `myevpnzone`, assign the previously created
+EVPN-controller and define 'node1' and 'node2' as exit nodes.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.3/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: myevpnzone
+VRF VXLAN Tag: 10000
+Controller: myevpnctl
+MTU: 1450
+VNet MAC Address: 32:F4:05:FE:6C:0A
+Exit Nodes: node1,node2
----
-Create a EVPN controller, using a private ASN number and above node addreesses
-as peers. Define 'node1' and 'node2' as gateway nodes.
+Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
----
-id: myevpnctl
-asn: 65000
-peers: 192.168.0.1,192.168.0.2,192.168.0.3
-gateway nodes: node1,node2
+ID: myvnet1
+Zone: myevpnzone
+Tag: 11000
----
-Create an EVPN zone named `myevpnzone' using the previously created
-EVPN-controller.
+Create a subnet on `myvnet1`:
----
-id: myevpnzone
-vrf vxlan tag: 10000
-controller: myevpnctl
-mtu: 1450
+Subnet: 10.0.1.0/24
+Gateway: 10.0.1.1
----
-Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone', a IPv4
-CIDR network and a random MAC address.
+Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
----
-id: myvnet1
-zone: myevpnzone
-tag: 11000
-ipv4: 10.0.1.1/24
-mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
+ID: myvnet2
+Zone: myevpnzone
+Tag: 12000
----
-Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
-different IPv4 CIDR network and a different random MAC address than `myvnet1'.
+Create a different subnet on `myvnet2``:
----
-id: myvnet2
-zone: myevpnzone
-tag: 12000
-ipv4: 10.0.2.1/24
-mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet
+Subnet: 10.0.2.0/24
+Gateway: 10.0.2.1
----
-Apply the configuration on the main SDN web-interface panel to create VNets
-locally on each nodes and generate the FRR config.
+Apply the configuration from the main SDN web-interface panel to create VNets
+locally on each node and generate the FRR configuration.
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
-
-Use the following network configuration for this VM:
+Use the following network configuration for 'vm1':
----
auto eth0
iface eth0 inet static
- address 10.0.1.100/24
- gateway 10.0.1.1 #this is the ip of the vnet1
- mtu 1450
+ address 10.0.1.100/24
+ gateway 10.0.1.1
+ mtu 1450
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
-`myvnet2'.
+Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
+`myvnet2`.
-Use the following network configuration for this VM:
+Use the following network configuration for 'vm2':
----
auto eth0
iface eth0 inet static
- address 10.0.2.100/24
- gateway 10.0.2.1 #this is the ip of the vnet2
- mtu 1450
+ address 10.0.2.100/24
+ gateway 10.0.2.1
+ mtu 1450
----
-Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
+Now you should be able to ping vm2 from vm1, and vm1 from vm2.
-If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
-will go to the configured 'myvnet2' gateway, then will be routed to gateway
+If you ping an external IP from 'vm2' on the non-gateway node3, the packet
+will go to the configured 'myvnet2' gateway, then will be routed to the exit
nodes ('node1' or 'node2') and from there it will leave those nodes over the
default gateway configured on node1 or node2.
-NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
-'10.0.2.0/24' network to node1, node2 on your external gateway, so that the
-public network can reply back.
+NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
+networks to node1 and node2 on your external gateway, so that the public network
+can reply back.
If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
and 10.0.2.0/24 in this example), will be announced dynamically.
+
+
+[[pvesdn_notes]]
+Notes
+-----
+
+Multiple EVPN Exit Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
+Reverse Path Filter) option, because packets can arrive at one node but go out
+from another node.
+
+Add the following to `/etc/sysctl.conf`:
+
+-----
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0
+-----
+
+VXLAN IPSEC Encryption
+~~~~~~~~~~~~~~~~~~~~~~
+
+To add IPSEC encryption on top of a VXLAN, this example shows how to use
+`strongswan`.
+
+You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
+IPv6 to handle encryption.
+
+So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
++ 50 (VXLAN) == 1500).
+
+Install strongswan on the host.
+
+----
+apt install strongswan
+----
+
+Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
+the VXLAN UDP port '4789'.
+
+----
+conn %default
+ ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
+ esp=aes256-sha1!
+ leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
+
+conn output
+ rightsubnet=%dynamic[udp/4789]
+ right=%any
+ type=transport
+ authby=psk
+ auto=route
+
+conn input
+ leftsubnet=%dynamic[udp/4789]
+ type=transport
+ authby=psk
+ auto=route
+----
+
+Generate a pre-shared key with:
+
+----
+openssl rand -base64 128
+----
+
+and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
+
+----
+: PSK <generatedbase64key>
+----
+
+Copy the PSK and the configuration to all nodes participating in the VXLAN network.
projects / pve-docs.git / blobdiff