[[chapter_pvesdn]]
-Software Defined Network
+Software-Defined Network
========================
ifndef::manvolnum[]
:pve-toplevel:
endif::manvolnum[]
-The SDN feature allow to create virtual networks (vnets)
-at datacenter level.
+The **S**oftware-**D**efined **N**etwork (SDN) feature in {pve} enables the
+creation of virtual zones and networks (VNets). This functionality simplifies
+advanced networking configurations and multitenancy setup.
-To enable SDN feature, you need to install "libpve-network-perl" package
+[[pvesdn_overview]]
+Introduction
+------------
+
+The {pve} SDN allows for separation and fine-grained control of virtual guest
+networks, using flexible, software-controlled configurations.
+
+Separation is managed through *zones*, virtual networks (*VNets*), and
+*subnets*. A zone is its own virtually separated network area. A VNet is a
+virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
+
+Depending on the type of the zone, the network behaves differently and offers
+specific features, advantages, and limitations.
+
+Use cases for SDN range from an isolated private network on each individual node
+to complex overlay networks across multiple PVE clusters on different locations.
+
+After configuring an VNet in the cluster-wide datacenter SDN administration
+interface, it is available as a common Linux bridge, locally on each node, to be
+assigned to VMs and Containers.
+
+
+[[pvesdn_support_status]]
+Support Status
+--------------
+
+History
+~~~~~~~
+
+The {pve} SDN stack has been available as an experimental feature since 2019 and
+has been continuously improved and tested by many developers and users.
+With its integration into the web interface in {pve} 6.2, a significant
+milestone towards broader integration was achieved.
+During the {pve} 7 release cycle, numerous improvements and features were added.
+Based on user feedback, it became apparent that the fundamental design choices
+and their implementation were quite sound and stable. Consequently, labeling it
+as `experimental' did not do justice to the state of the SDN stack.
+For {pve} 8, a decision was made to lay the groundwork for full integration of
+the SDN feature by elevating the management of networks and interfaces to a core
+component in the {pve} access control stack.
+In {pve} 8.1, two major milestones were achieved: firstly, DHCP integration was
+added to the IP address management (IPAM) feature, and secondly, the SDN
+integration is now installed by default.
+
+Current Status
+~~~~~~~~~~~~~~
+
+The current support status for the various layers of our SDN installation is as
+follows:
+
+- Core SDN, which includes VNet management and its integration with the {pve}
+ stack, is fully supported.
+- IPAM, including DHCP management for virtual guests, is in tech preview.
+- Complex routing via FRRouting and controller integration are in tech preview.
+
+[[pvesdn_installation]]
+Installation
+------------
+
+SDN Core
+~~~~~~~~
+
+Since {pve} 8.1 the core Software-Defined Network (SDN) packages are installed
+by default.
+
+If you upgrade from an older version, you need to install the
+`libpve-network-perl` package on every node:
----
+apt update
apt install libpve-network-perl
----
-A vnet is a bridge with a vlan or vxlan tag.
+NOTE: {pve} version 7.0 and above have the `ifupdown2` package installed by
+default. If you originally installed your system with an older version, you need
+to explicitly install the `ifupdown2` package.
+
+After installation, you need to ensure that the following line is present at the
+end of the `/etc/network/interfaces` configuration file on all nodes, so that
+the SDN configuration gets included and activated.
+
+----
+source /etc/network/interfaces.d/*
+----
-The vnets are deployed locally on each node after configuration
-commit at datacenter level.
+[[pvesdn_install_dhcp_ipam]]
+DHCP IPAM
+~~~~~~~~~
-You need to have "ifupdown2" package installed on each node to manage local
-configuration reloading.
+The DHCP integration into the built-in 'PVE' IP Address Management stack
+currently uses `dnsmasq` for giving out DHCP leases. This is currently opt-in.
+
+To use that feature you need to install the `dnsmasq` package on every node:
----
-apt install ifupdown2
+apt update
+apt install dnsmasq
+# disable default instance
+systemctl disable --now dnsmasq
----
-Main configuration
-------------------
+[[pvesdn_install_frrouting]]
+FRRouting
+~~~~~~~~~
+
+The {pve} SDN stack uses the https://frrouting.org/[FRRouting] project for
+advanced setups. This is currently opt-in.
+
+To use the SDN routing integration you need to install the `frr-pythontools`
+package on all nodes:
+
+----
+apt update
+apt install frr-pythontools
+----
-The configuration is done at datacenter level.
+[[pvesdn_main_configuration]]
+Configuration Overview
+----------------------
-The sdn feature have 4 main sections for the configuration
+Configuration is done at the web UI at datacenter level, separated into the
+following sections:
-* SDN
+* SDN:: Here you get an overview of the current active SDN state, and you can
+ apply all pending changes to the whole cluster.
-* Zones
+* xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
+ network zones
-* Vnets
+* xref:pvesdn_config_vnet[VNets] VNets: Create virtual network bridges and
+ manage subnets
-* Controller
+The Options category allows adding and managing additional services to be used
+in your SDN setup.
+* xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
+ in complex setups
-SDN
-~~~
+* DHCP: Define a DHCP server for a zone that automatically allocates IPs for
+ guests in the IPAM and leases them to the guests via DHCP.
-[thumbnail="screenshot/gui-sdn-status.png"]
+* xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
+ guests
-This is the Main panel, where you can see deployment of zones on differents nodes.
+* xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
+ virtual guests' hostname and IP addresses
-They are an "apply" button, to push && reload local configuration on differents nodes.
+[[pvesdn_tech_and_config_overview]]
+Technology & Configuration
+--------------------------
+The {pve} Software-Defined Network implementation uses standard Linux networking
+as much as possible. The reason for this is that modern Linux networking
+provides almost all needs for a feature full SDN implementation and avoids adding
+external dependencies and reduces the overall amount of components that can
+break.
+The {pve} SDN configurations are located in `/etc/pve/sdn`, which is shared with
+all other cluster nodes through the {pve} xref:chapter_pmxcfs[configuration file system].
+Those configurations get translated to the respective configuration formats of
+the tools that manage the underlying network stack (for example `ifupdown2` or
+`frr`).
+
+New changes are not immediately applied but recorded as pending first. You can
+then apply a set of different changes all at once in the main 'SDN' overview
+panel on the web interface. This system allows to roll-out various changes as
+single atomic one.
+
+The SDN tracks the rolled-out state through the '.running-config' and '.version'
+files located in '/etc/pve/sdn'.
+
+// TODO: extend implementation and technology details.
+
+[[pvesdn_config_zone]]
Zones
-~~~~~
+-----
-[thumbnail="screenshot/gui-sdn-zone.png"]
+A zone defines a virtually separated network. Zones are restricted to
+specific nodes and assigned permissions, in order to restrict users to a certain
+zone and its contained VNets.
-A zone will defined the kind of virtual network you want to defined.
+Different technologies can be used for separation:
-it can be
+* Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
-* vlan
+* VLAN: Virtual LANs are the classic method of subdividing a LAN
-* QinQ (stacked vlan)
+* QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
-* vxlan (layer2 vxlan)
+* VXLAN: Layer 2 VXLAN network via a UDP tunnel
-* bgp-evpn (vxlan with layer3 routing)
+* EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
-You can restrict a zone to specific nodes.
-It's also possible to add permissions on a zone, to restrict user
-to use only a specific zone and the vnets in this zone
+[[pvesdn_config_common_options]]
+Common Options
+~~~~~~~~~~~~~~
-Vnets
-~~~~~
+The following options are available for all zone types:
-[thumbnail="screenshot/gui-sdn-vnet-evpn.png"]
+Nodes:: The nodes which the zone and associated VNets should be deployed on.
-A vnet is a bridge that will be deployed locally on the node,
-for vm communication. (Like a classic vmbrX).
+IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
+ zone. Optional, defaults to `pve`.
-Vnet properties are:
+DNS:: DNS API server. Optional.
-* ID: a 8 characters ID
+ReverseDNS:: Reverse DNS API server. Optional.
-* Alias: Optionnal bigger name
+DNSZone:: DNS domain name. Used to register hostnames, such as
+ `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
-* Zone: The associated zone of the vnet
-* Tag: unique vlan or vxlan id
+[[pvesdn_zone_plugin_simple]]
+Simple Zones
+~~~~~~~~~~~~
-* ipv4: an anycast ipv4 address (same bridge ip deployed on each node), for bgp-evpn routing only
+This is the simplest plugin. It will create an isolated VNet bridge. This
+bridge is not linked to a physical interface, and VM traffic is only local on
+each the node.
+It can be used in NAT or routed setups.
-* ipv6: an anycast ipv6 address (same bridge ip deployed on each node), for bgp-evpn routing only
+[[pvesdn_zone_plugin_vlan]]
+VLAN Zones
+~~~~~~~~~~
-Controllers
+The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
+node's physical interface. It uses VLAN tagging defined in the VNet to isolate
+the network segments. This allows connectivity of VMs between different nodes.
+
+VLAN zone configuration options:
+
+Bridge:: The local bridge or OVS switch, already configured on *each* node that
+ allows node-to-node connection.
+
+
+[[pvesdn_zone_plugin_qinq]]
+QinQ Zones
+~~~~~~~~~~
+
+QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
+isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
+whereas the inner VLAN tag is defined by the VNet.
+
+NOTE: Your physical network switches must support stacked VLANs for this
+configuration.
+
+QinQ zone configuration options:
+
+Bridge:: A local, VLAN-aware bridge that is already configured on each local
+ node
+
+Service VLAN:: The main VLAN tag of this zone
+
+Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
+ 802.1ad service VLAN type.
+
+MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
+ For example, you must reduce the MTU to `1496` if you physical interface MTU is
+ `1500`.
+
+
+[[pvesdn_zone_plugin_vxlan]]
+VXLAN Zones
~~~~~~~~~~~
-[thumbnail="screenshot/gui-sdn-controller.png"]
+The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
+(underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
+datagrams using the default destination port `4789`.
+
+You have to configure the underlay network yourself to enable UDP connectivity
+between all peers.
+
+You can, for example, create a VXLAN overlay network on top of public internet,
+appearing to the VMs as if they share the same local Layer 2 network.
-Some zone plugins (Currently : bgp-evpn only),
-need an external controller to manage the vnets control-plane.
+WARNING: VXLAN on its own does does not provide any encryption. When joining
+ multiple sites via VXLAN, make sure to establish a secure connection between
+ the site, for example by using a site-to-site VPN.
+VXLAN zone configuration options:
+Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
+ can be external nodes reachable at this IP address.
+ All nodes in the cluster need to be mentioned here.
-Zones Plugins
--------------
-common zone options:
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ lower than the outgoing physical interface.
-* nodes: restrict deploy of the vnets of theses nodes only
+[[pvesdn_zone_plugin_evpn]]
+EVPN Zones
+~~~~~~~~~~
-Vlan
-~~~~~
+The EVPN zone creates a routable Layer 3 network, capable of spanning across
+multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
+the routing protocol.
-[thumbnail="screenshot/gui-sdn-zone-vlan.png"]
+The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
+IP is the same on each node, meaning a virtual guest can use this address as
+gateway.
-This is the most simple plugin, it'll reuse an existing local bridge or ovs,
-and manage vlan on it.
-The benefit of using sdn module, is that you can create different zones with specific
-vnets vlan tag, and restrict your customers on their zones.
+Routing can work across VNets from different zones through a VRF (Virtual
+Routing and Forwarding) interface.
-specific qinq configuration options:
+EVPN zone configuration options:
-* bridge: a local vlan-aware bridge or ovs switch already configured on each local node
+VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
+ It must be different than the VXLAN-ID of the VNets.
-QinQ
-~~~~~
+Controller:: The EVPN-controller to use for this zone. (See controller plugins
+ section).
-[thumbnail="screenshot/gui-sdn-zone-qinq.png"]
+VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
+ zone. Will be auto-generated if not defined.
-QinQ is stacked vlan.
-you have the first vlan tag defined on the zone (service-vlan), and
-the second vlan tag defined on the vnets
+Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
+ network, through the real network. The configured nodes will announce a
+ default route in the EVPN network. Optional.
-Your physical network switchs need to support stacked vlans !
+Primary Exit Node:: If you use multiple exit nodes, force traffic through this
+ primary exit node, instead of load-balancing on all nodes. Optional but
+ necessary if you want to use SNAT or if your upstream router doesn't support
+ ECMP.
-specific qinq configuration options:
+Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
+ service from an exit node. (By default, the exit nodes only allow forwarding
+ traffic between real network and EVPN network). Optional.
-* bridge: a local vlan-aware bridge already configured on each local node
-* service vlan: The main vlan tag of this zone
-* mtu: you need 4 more bytes for the double tag vlan.
- You can reduce the mtu to 1496 if you physical interface mtu is 1500.
+Advertise Subnets:: Announce the full subnet in the EVPN network.
+ If you have silent VMs/CTs (for example, if you have multiple IPs and the
+ anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
+ able to be reached inside the EVPN network). Optional.
-Vxlan
-~~~~~
+Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
+ packets. This is required if you use floating IPs in your VMs (IP and MAC
+ addresses are being moved between systems). Optional.
-[thumbnail="screenshot/gui-sdn-zone-vxlan.png"]
+Route-target Import:: Allows you to import a list of external EVPN route
+ targets. Used for cross-DC or different EVPN network interconnects. Optional.
-The vxlan plugin will established vxlan tunnel (overlay) on top of an existing network (underlay).
-you can for example, create a private ipv4 vxlan network on top of public internet network nodes.
-This is a layer2 tunnel only, no routing between different vnets is possible.
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ less than the maximal MTU of the outgoing physical interface. Optional,
+ defaults to 1450.
-Each vnet will have a specific vxlan id ( 1 - 16777215 )
+[[pvesdn_config_vnet]]
+VNets
+-----
-Specific evpn configuration options:
+After creating a virtual network (VNet) through the SDN GUI, a local network
+interface with the same name is available on each node. To connect a guest to the
+VNet, assign the interface to the guest and set the IP address accordingly.
-* peers address list: an ip list of all nodes where you want to communicate (could be also external nodes)
+Depending on the zone, these options have different meanings and are explained
+in the respective zone section in this document.
-* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
- than the outgoing physical interface.
+WARNING: In the current state, some options may have no effect or won't work in
+certain zones.
-evpn
-~~~~
+VNet configuration options:
-[thumbnail="screenshot/gui-sdn-zone-evpn.png"]
+ID:: An up to 8 character ID to identify a VNet
-This is the most complex plugin.
+Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
-BGP-evpn allow to create routable layer3 network.
-The vnet of evpn can have an anycast ip address/mac address.
-The bridge ip is the same on each node, then vm can use
-as gateway.
-The routing is working only across vnets of a specific zone through a vrf.
+Zone:: The associated zone for this VNet
-Specific evpn configuration options:
+Tag:: The unique VLAN or VXLAN ID
-* vrf vxlan tag: This is a vxlan-id used for routing interconnect between vnets,
- it must be different than vxlan-id of vnets
+VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
+ in the guest.
-* controller: an evpn need to be defined first (see controller plugins section)
-* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
- than the outgoing physical interface.
+[[pvesdn_config_subnet]]
+Subnets
+-------
+A subnet define a specific IP range, described by the CIDR network address.
+Each VNet, can have one or more subnets.
-Controllers Plugins
--------------------
+A subnet can be used to:
-evpn
-~~~~
+* Restrict the IP addresses you can define on a specific VNet
+* Assign routes/gateways on a VNet in layer 3 zones
+* Enable SNAT on a VNet in layer 3 zones
+* Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
+* DNS registration through DNS plugins
-[thumbnail="screenshot/gui-sdn-controller-evpn.png"]
+If an IPAM server is associated with the subnet zone, the subnet prefix will be
+automatically registered in the IPAM.
-For bgp-evpn, we need a controller to manage the control plane.
-The software controller is "frr" router.
-You need to install it on each node where you want to deploy the evpn zone.
+Subnet configuration options:
+
+ID:: A CIDR network address, for example 10.0.0.0/8
+
+Gateway:: The IP address of the network's default gateway. On layer 3 zones
+ (Simple/EVPN plugins), it will be deployed on the VNet.
+
+SNAT:: Enable Source NAT which allows VMs from inside a
+ VNet to connect to the outside network by forwarding the packets to the nodes
+ outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
+ Optional.
+
+DNS Zone Prefix:: Add a prefix to the domain registration, like
+ <hostname>.prefix.<domain> Optional.
+
+
+[[pvesdn_config_controllers]]
+Controllers
+-----------
+
+Some zones implement a separated control and data plane that require an external
+controller to manage the VNet's control plane.
+
+Currently, only the `EVPN` zone requires an external controller.
+
+
+[[pvesdn_controller_plugin_evpn]]
+EVPN Controller
+~~~~~~~~~~~~~~~
+
+The `EVPN`, zone requires an external controller to manage the control plane.
+The EVPN controller plugin configures the Free Range Routing (frr) router.
+
+To enable the EVPN controller, you need to install frr on every node that shall
+participate in the EVPN zone.
----
-apt install frr
+apt install frr frr-pythontools
----
-configuration options:
+EVPN controller configuration options:
-*asn: a unique bgp asn number.
- It's recommended to use private asn number (64512 – 65534, 4200000000 – 4294967294)
+ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
+ breaking global routing by mistake.
-*peers: an ip list of all nodes where you want to communicate (could be also external nodes or route reflectors servers)
+Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
+ external nodes or route reflector servers)
-If you want to route traffic from the sdn bgp-evpn network to external world:
-* gateway-nodes: The proxmox nodes from where the bgp-evpn traffic will exit to external through the nodes default gateway
+[[pvesdn_controller_plugin_BGP]]
+BGP Controller
+~~~~~~~~~~~~~~
-If you want that gateway nodes don't use the default gateway, but for example, sent traffic to external bgp routers
+The BGP controller is not used directly by a zone.
+You can use it to configure FRR to manage BGP peers.
-* gateway-external-peers: 192.168.0.253,192.168.0.254
+For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
+It can also be used to export EVPN routes to an external BGP peer.
+NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
+controller.
-Local deployment Monitoring
----------------------------
+BGP controller configuration options:
-[thumbnail="screenshot/gui-sdn-local-status.png"]
+Node:: The node of this BGP controller
-After apply configuration on the main sdn section,
-the local configuration is generated locally on each node,
-in /etc/network/interfaces.d/sdn, and reloaded.
+ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
+ you could break global routing by mistake.
-You can monitor the status of local zones && vnets through the main tree.
+Peer:: A list of peer IP addresses you want to communicate with using the
+ underlying BGP network.
+EBGP:: If your peer's remote-AS is different, this enables EBGP.
+Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
-Vlan setup example
-------------------
-node1: /etc/network/interfaces
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
+ not directly connected or they use loopback.
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.1/24
+bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
-source /etc/network/interfaces.d/*
-----
+[[pvesdn_controller_plugin_ISIS]]
+ISIS Controller
+~~~~~~~~~~~~~~~
-node2: /etc/network/interfaces
+The ISIS controller is not used directly by a zone.
+You can use it to configure FRR to export EVPN routes to an ISIS domain.
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+ISIS controller configuration options:
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.2/24
+Node:: The node of this ISIS controller.
-source /etc/network/interfaces.d/*
-----
+Domain:: A unique ISIS domain.
-create an vlan zone
+Network Entity Title:: A Unique ISIS network address that identifies this node.
-----
-id: mylanzone
-bridge: vmbr0
-----
+Interfaces:: A list of physical interface(s) used by ISIS.
-create a vnet1 with vlan-id 10
+Loopback:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
-----
-id: myvnet1
-zone: myvlanzone
-tag: 10
+
+[[pvesdn_config_ipam]]
+IPAM
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+IP Address Management (IPAM) tools manage the IP addresses of clients on the
+network. SDN in {pve} uses IPAM for example to find free IP addresses for new
+guests.
+A single IPAM instance can be associated with one or more zones.
-create a vm1, with 1 nic on vnet1 on node1
-----
-auto eth0
-iface eth0 inet static
- address 10.0.3.100/24
-----
+[[pvesdn_ipam_plugin_pveipam]]
+PVE IPAM Plugin
+~~~~~~~~~~~~~~~
+
+The default built-in IPAM for your {pve} cluster.
+
+You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in
+the SDN section of the datacenter configuration. This UI can be used to create,
+update and delete IP mappings. This is particularly convenient in conjunction
+with the xref:pvesdn_config_dhcp[DHCP feature].
+
+If you are using DHCP, you can use the IPAM panel to create or edit leases for
+specific VMs, which enables you to change the IPs allocated via DHCP. When
+editing an IP of a VM that is using DHCP you must make sure to force the guest
+to acquire a new DHCP leases. This can usually be done by reloading the network
+stack of the guest or rebooting it.
+
+[[pvesdn_ipam_plugin_netbox]]
+NetBox IPAM Plugin
+~~~~~~~~~~~~~~~~~~
+
+link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
+Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
+
+To integrate NetBox with {pve} SDN, create an API token in NetBox as described
+here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
+
+The NetBox configuration properties are:
+
+URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
+
+Token:: An API access token
+
+
+[[pvesdn_ipam_plugin_phpipam]]
+phpIPAM Plugin
+~~~~~~~~~~~~~~
+
+In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
+an API token with admin privileges to the application.
+
+The phpIPAM configuration properties are:
+
+URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
+
+Token:: An API access token
+
+Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
+ installations use `sectionid=1` for customers.
+
+
+[[pvesdn_config_dns]]
+DNS
+---
+
+The DNS plugin in {pve} SDN is used to define a DNS API server for registration
+of your hostname and IP address. A DNS configuration is associated with one or
+more zones, to provide DNS registration for all the subnet IPs configured for
+a zone.
+
+[[pvesdn_dns_plugin_powerdns]]
+PowerDNS Plugin
+~~~~~~~~~~~~~~~
+https://doc.powerdns.com/authoritative/http-api/index.html
+
+You need to enable the web server and the API in your PowerDNS config:
-create a vm2, with 1 nic on vnet1 on node2
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.101/24
+api=yes
+api-key=arandomgeneratedstring
+webserver=yes
+webserver-port=8081
----
-Then, you should be able to ping between between vm1 && vm2
+The PowerDNS configuration options are:
+url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
-QinQ setup example
-------------------
-node1: /etc/network/interfaces
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+key:: An API access key
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.1/24
+ttl:: The default TTL for records
-source /etc/network/interfaces.d/*
-----
-node2: /etc/network/interfaces
+[[pvesdn_config_dhcp]]
+DHCP
+------
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+The DHCP plugin in {pve} SDN can be used to automatically deploy a DHCP server
+for a Zone. It provides DHCP for all Subnets in a Zone that have a DHCP range
+configured. Currently the only available backend plugin for DHCP is the dnsmasq
+plugin.
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.2/24
+The DHCP plugin works by allocating an IP in the IPAM plugin configured in the
+Zone when adding a new network interface to a VM/CT. You can find more
+information on how to configure an IPAM in the
+xref:pvesdn_config_ipam[respective section of our documentation].
+
+When the VM starts, a mapping for the MAC address and IP gets created in the DHCP
+plugin of the zone. When the network interfaces is removed or the VM/CT are
+destroyed, then the entry in the IPAM and the DHCP server are deleted as well.
+
+NOTE: Some features (adding/editing/removing IP mappings) are currently only
+available when using the xref:pvesdn_ipam_plugin_pveipam[PVE IPAM plugin].
-source /etc/network/interfaces.d/*
-----
-create an qinq zone1 with service vlan 20
+Configuration
+~~~~~~~~~~~~~
+
+You can enable automatic DHCP for a zone in the Web UI via the Zones panel and
+enabling DHCP in the advanced options of a zone.
+
+NOTE: Currently only Simple Zones have support for automatic DHCP
+
+After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be
+configured for the subnets in a Zone. In order to that, go to the Vnets panel and
+select the Subnet for which you want to configure DHCP ranges. In the edit
+dialogue you can configure DHCP ranges in the respective Tab. Alternatively you
+can set DHCP ranges for a Subnet via the following CLI command:
----
-id: qinqzone1
-bridge: vmbr0
-service vlan: 20
+pvesh set /cluster/sdn/vnets/<vnet>/subnets/<subnet>
+ -dhcp-range start-address=10.0.1.100,end-address=10.0.1.200
+ -dhcp-range start-address=10.0.2.100,end-address=10.0.2.200
----
-create an qinq zone2 with service vlan 30
+You also need to have a gateway configured for the subnet - otherwise
+automatic DHCP will not work.
+
+The DHCP plugin will then allocate IPs in the IPAM only in the configured
+ranges.
+
+Do not forget to follow the installation steps for the
+xref:pvesdn_install_dhcp_ipam[dnsmasq DHCP plugin] as well.
+
+Plugins
+~~~~~~~
+
+Dnsmasq Plugin
+^^^^^^^^^^^^^^
+Currently this is the only DHCP plugin and therefore the plugin that gets used
+when you enable DHCP for a zone.
+
+.Installation
+For installation see the xref:pvesdn_install_dhcp_ipam[DHCP IPAM] section.
+
+.Configuration
+The plugin will create a new systemd service for each zone that dnsmasq gets
+deployed to. The name for the service is `dnsmasq@<zone>`. The lifecycle of this
+service is managed by the DHCP plugin.
+
+The plugin automatically generates the following configuration files in the
+folder `/etc/dnsmasq.d/<zone>`:
+
+`00-default.conf`::
+This contains the default global configuration for a dnsmasq instance.
+
+`10-<zone>-<subnet_cidr>.conf`::
+This file configures specific options for a subnet, such as the DNS server that
+should get configured via DHCP.
+
+`10-<zone>-<subnet_cidr>.ranges.conf`::
+This file configures the DHCP ranges for the dnsmasq instance.
+
+`ethers`::
+This file contains the MAC-address and IP mappings from the IPAM plugin. In
+order to override those mappings, please use the respective IPAM plugin rather
+than editing this file, as it will get overwritten by the dnsmasq plugin.
+
+You must not edit any of the above files, since they are managed by the DHCP
+plugin. In order to customize the dnsmasq configuration you can create
+additional files (e.g. `90-custom.conf`) in the configuration folder - they will
+not get changed by the dnsmasq DHCP plugin.
+
+Configuration files are read in order, so you can control the order of the
+configuration directives by naming your custom configuration files appropriately.
+
+DHCP leases are stored in the file `/var/lib/misc/dnsmasq.<zone>.leases`.
+
+When using the PVE IPAM plugin, you can update, create and delete DHCP leases.
+For more information please consult the documentation of
+xref:pvesdn_ipam_plugin_pveipam[the PVE IPAM plugin]. Changing DHCP leases is
+currently not supported for the other IPAM plugins.
+
+[[pvesdn_setup_examples]]
+Examples
+--------
+
+This section presents multiple configuration examples tailored for common SDN
+use cases. It aims to offer tangible implementations, providing additional
+details to enhance comprehension of the available configuration options.
+
+
+[[pvesdn_setup_example_simple]]
+Simple Zone Example
+~~~~~~~~~~~~~~~~~~~
+
+Simple zone networks create an isolated network for guests on a single host to
+connect to each other.
+
+TIP: connection between guests are possible if all guests reside on a same host
+but cannot be reached on other nodes.
+
+* Create a simple zone named `simple`.
+* Add a VNet names `vnet1`.
+* Create a Subnet with a gateway and the SNAT option enabled.
+* This creates a network bridge `vnet1` on the node. Assign this bridge to the
+ guests that shall join the network and configure an IP address.
+
+The network interface configuration in two VMs may look like this which allows
+them to communicate via the 10.0.1.0/24 network.
----
-id: qinqzone2
-bridge: vmbr0
-service vlan: 30
+allow-hotplug ens19
+iface ens19 inet static
+ address 10.0.1.14/24
----
-create a vnet1 with customer vlan-id 100 on qinqzone1
-
----
-id: myvnet1
-zone: qinqzone1
-tag: 100
+allow-hotplug ens19
+iface ens19 inet static
+ address 10.0.1.15/24
----
-create a vnet2 with customer vlan-id 100 on qinqzone2
+
+[[pvesdn_setup_example_nat]]
+Source NAT Example
+~~~~~~~~~~~~~~~~~~
+
+If you want to allow outgoing connections for guests in the simple network zone
+the simple zone offers a Source NAT (SNAT) option.
+
+Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
+Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
----
-id: myvnet2
-zone: qinqzone1
-tag: 100
+Subnet: 172.16.0.0/24
+Gateway: 172.16.0.1
+SNAT: checked
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+In the guests configure the static IP address inside the subnet's IP range.
+
+The node itself will join this network with the Gateway IP '172.16.0.1' and
+function as the NAT gateway for guests within the subnet range.
-create a vm1, with 1 nic on vnet1 on node1
+[[pvesdn_setup_example_vlan]]
+VLAN Setup Example
+~~~~~~~~~~~~~~~~~~
+
+When VMs on different nodes need to communicate through an isolated network, the
+VLAN zone allows network level isolation using VLAN tags.
+
+Create a VLAN zone named `myvlanzone`:
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.100/24
+ID: myvlanzone
+Bridge: vmbr0
----
-create a vm2, with 1 nic on vnet1 on node2
+Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
+`myvlanzone`.
+
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.101/24
+ID: myvnet1
+Zone: myvlanzone
+Tag: 10
----
-create a vm3, with 1 nic on vnet2 on node1
+Apply the configuration through the main SDN panel, to create VNets locally on
+each node.
+
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
+
+Use the following network configuration for this VM:
----
auto eth0
iface eth0 inet static
- address 10.0.3.102/24
+ address 10.0.3.100/24
----
-create a vm4, with 1 nic on vnet2 on node2
+Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
+`myvnet1` as vm1.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
- address 10.0.3.103/24
+ address 10.0.3.101/24
----
-Then, you should be able to ping between between vm1 && vm2
-vm3 && vm4 could ping together
+Following this, you should be able to ping between both VMs using that network.
+
+
+[[pvesdn_setup_example_qinq]]
+QinQ Setup Example
+~~~~~~~~~~~~~~~~~~
-but vm1 && vm2 couldn't ping vm3 && vm4,
-as it's a different zone, with different service vlan
+This example configures two QinQ zones and adds two VMs to each zone to
+demonstrate the additional layer of VLAN tags which allows the configuration of
+more isolated VLANs.
-Vxlan setup example
--------------------
-node1: /etc/network/interfaces
+A typical use case for this configuration is a hosting provider that provides an
+isolated network to customers for VM communication but isolates the VMs from
+other customers.
+
+Create a QinQ zone named `qinqzone1` with service VLAN 20
+
+----
+ID: qinqzone1
+Bridge: vmbr0
+Service VLAN: 20
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.1/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-source /etc/network/interfaces.d/*
+Create another QinQ zone named `qinqzone2` with service VLAN 30
+----
+ID: qinqzone2
+Bridge: vmbr0
+Service VLAN: 30
----
-node2: /etc/network/interfaces
+Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
+`qinqzone1` zone.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.2/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: qinqvnet1
+Zone: qinqzone1
+Tag: 100
----
-node3: /etc/network/interfaces
+Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.3/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+ID: qinqvnet2
+Zone: qinqzone2
+Tag: 100
+----
+
+Apply the configuration on the main SDN web interface panel to create VNets
+locally on each node.
+
+Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
+interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
+`qinqvnet2`.
+
+Inside the VM, configure the IP addresses of the interfaces, for example via
+`/etc/network/interfaces`:
-source /etc/network/interfaces.d/*
----
+auto eth0
+iface eth0 inet static
+ address 10.0.3.101/24
+----
+// TODO: systemd-network example
+Configure all four VMs to have IP addresses from the '10.0.3.101' to
+'10.0.3.104' range.
+
+Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
+between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
+'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
-create an vxlan zone
+
+[[pvesdn_setup_example_vxlan]]
+VXLAN Setup Example
+~~~~~~~~~~~~~~~~~~~
+
+The example assumes a cluster with three nodes, with the node IP addresses
+192.168.0.1, 192.168.0.2 and 192.168.0.3.
+
+Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
+peer address list. Use the default MTU of 1450 or configure accordingly.
----
-id: myvxlanzone
-peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
-mtu: 1450
+ID: myvxlanzone
+Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
----
-create first vnet
+Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
+previously.
----
-id: myvnet1
-zone: myvxlanzone
-tag: 100000
+ID: vxvnet1
+Zone: myvxlanzone
+Tag: 100000
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+Apply the configuration on the main SDN web interface panel to create VNets
+locally on each nodes.
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
-create a vm1, with 1 nic on vnet1 on node2
+Use the following network configuration for this VM (note the lower MTU).
----
auto eth0
iface eth0 inet static
- address 10.0.3.100/24
- mtu 1450
+ address 10.0.3.100/24
+ mtu 1450
----
-create a vm2, with 1 nic on vnet1 on node3
+Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
+`vxvnet1` as vm1.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
- address 10.0.3.101/24
- mtu 1450
+ address 10.0.3.101/24
+ mtu 1450
----
-Then, you should be able to ping between between vm1 && vm2
-
+Then, you should be able to ping between between 'vm1' and 'vm2'.
-EVPN setup example
-------------------
-node1: /etc/network/interfaces
+[[pvesdn_setup_example_evpn]]
+EVPN Setup Example
+~~~~~~~~~~~~~~~~~~
-----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.1/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
-----
+The example assumes a cluster with three nodes (node1, node2, node3) with IP
+addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
-node2: /etc/network/interfaces
+Create an EVPN controller, using a private ASN number and the above node
+addresses as peers.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.2/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: myevpnctl
+ASN#: 65000
+Peers: 192.168.0.1,192.168.0.2,192.168.0.3
----
-node3: /etc/network/interfaces
+Create an EVPN zone named `myevpnzone`, assign the previously created
+EVPN-controller and define 'node1' and 'node2' as exit nodes.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.3/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: myevpnzone
+VRF VXLAN Tag: 10000
+Controller: myevpnctl
+MTU: 1450
+VNet MAC Address: 32:F4:05:FE:6C:0A
+Exit Nodes: node1,node2
----
-create a evpn controller
+Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
----
-id: myevpnctl
-asn: 65000
-peers: 192.168.0.1,192.168.0.2,192.168.0.3
-gateway nodes: node1,node2
+ID: myvnet1
+Zone: myevpnzone
+Tag: 11000
----
-create an evpn zone
+Create a subnet on `myvnet1`:
----
-id: myevpnzone
-vrf vxlan tag: 10000
-controller: myevpnctl
-mtu: 1450
+Subnet: 10.0.1.0/24
+Gateway: 10.0.1.1
----
-create first vnet
+Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
----
-id: myvnet1
-zone: myevpnzone
-tag: 11000
-ipv4: 10.0.1.1/24
-mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
+ID: myvnet2
+Zone: myevpnzone
+Tag: 12000
----
-create second vnet
+Create a different subnet on `myvnet2``:
----
-id: myvnet2
-zone: myevpnzone
-tag: 12000
-ipv4: 10.0.2.1/24
-mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet
+Subnet: 10.0.2.0/24
+Gateway: 10.0.2.1
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
-
+Apply the configuration from the main SDN web interface panel to create VNets
+locally on each node and generate the FRR configuration.
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
-create a vm1, with 1 nic on vnet1 on node2
+Use the following network configuration for 'vm1':
----
auto eth0
iface eth0 inet static
- address 10.0.1.100/24
- gateway 10.0.1.1 #this is the ip of the vnet1
- mtu 1450
+ address 10.0.1.100/24
+ gateway 10.0.1.1
+ mtu 1450
----
-create a vm2, with 1 nic on vnet2 on node3
+Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
+`myvnet2`.
+
+Use the following network configuration for 'vm2':
+
----
auto eth0
iface eth0 inet static
- address 10.0.2.100/24
- gateway 10.0.2.1 #this is the ip of the vnet2
- mtu 1450
+ address 10.0.2.100/24
+ gateway 10.0.2.1
+ mtu 1450
+----
+
+
+Now you should be able to ping vm2 from vm1, and vm1 from vm2.
+
+If you ping an external IP from 'vm2' on the non-gateway node3, the packet
+will go to the configured 'myvnet2' gateway, then will be routed to the exit
+nodes ('node1' or 'node2') and from there it will leave those nodes over the
+default gateway configured on node1 or node2.
+
+NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
+networks to node1 and node2 on your external gateway, so that the public network
+can reply back.
+
+If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
+and 10.0.2.0/24 in this example), will be announced dynamically.
+
+
+[[pvesdn_notes]]
+Notes
+-----
+
+Multiple EVPN Exit Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
+Reverse Path Filter) option, because packets can arrive at one node but go out
+from another node.
+
+Add the following to `/etc/sysctl.conf`:
+
+-----
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0
+-----
+
+VXLAN IPSEC Encryption
+~~~~~~~~~~~~~~~~~~~~~~
+
+To add IPSEC encryption on top of a VXLAN, this example shows how to use
+`strongswan`.
+
+You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
+IPv6 to handle encryption.
+
+So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
++ 50 (VXLAN) == 1500).
+
+Install strongswan on the host.
+
+----
+apt install strongswan
----
+Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
+the VXLAN UDP port '4789'.
-Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
+----
+conn %default
+ ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
+ esp=aes256-sha1!
+ leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
+
+conn output
+ rightsubnet=%dynamic[udp/4789]
+ right=%any
+ type=transport
+ authby=psk
+ auto=route
-from vm2 on node3, if you ping an external ip, the packet will go
-to the vnet2 gateway, then will be routed to gateway nodes (node1 or node2)
-then the packet will be routed to the node1 or node2 default gw.
+conn input
+ leftsubnet=%dynamic[udp/4789]
+ type=transport
+ authby=psk
+ auto=route
+----
-Of course you need to add reverse routes to 10.0.1.0/24 && 10.0.2.0/24 to node1,node2 on your external gateway.
+Generate a pre-shared key with:
-If you have configured an external bgp router, the bgp-evpn routes (10.0.1.0/24 && 10.0.2.0/24),
-will be announced dynamically.
+----
+openssl rand -base64 128
+----
+
+and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
+
+----
+: PSK <generatedbase64key>
+----
+Copy the PSK and the configuration to all nodes participating in the VXLAN network.
projects / pve-docs.git / blobdiff