:pve-toplevel:
endif::manvolnum[]
-The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
-virtual networks (VNets) at the datacenter level.
+The **S**oftware-**D**efined **N**etwork (SDN) feature in {pve} enables the
+creation of virtual zones and networks (VNets). This functionality simplifies
+advanced networking configurations and multitenancy setup.
+
+[[pvesdn_overview]]
+Introduction
+------------
+
+The {pve} SDN allows for separation and fine-grained control of virtual guest
+networks, using flexible, software-controlled configurations.
+
+Separation is managed through *zones*, virtual networks (*VNets*), and
+*subnets*. A zone is its own virtually separated network area. A VNet is a
+virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
+
+Depending on the type of the zone, the network behaves differently and offers
+specific features, advantages, and limitations.
+
+Use cases for SDN range from an isolated private network on each individual node
+to complex overlay networks across multiple PVE clusters on different locations.
+
+After configuring an VNet in the cluster-wide datacenter SDN administration
+interface, it is available as a common Linux bridge, locally on each node, to be
+assigned to VMs and Containers.
-WARNING: SDN is currently an **experimental feature** in {pve}. This
-documentation for it is also still under development. Ask on our
-xref:getting_help[mailing lists or in the forum] for questions and feedback.
+[[pvesdn_support_status]]
+Support Status
+--------------
+
+History
+~~~~~~~
+
+The {pve} SDN stack has been available as an experimental feature since 2019 and
+has been continuously improved and tested by many developers and users.
+With its integration into the web interface in {pve} 6.2, a significant
+milestone towards broader integration was achieved.
+During the {pve} 7 release cycle, numerous improvements and features were added.
+Based on user feedback, it became apparent that the fundamental design choices
+and their implementation were quite sound and stable. Consequently, labeling it
+as `experimental' did not do justice to the state of the SDN stack.
+For {pve} 8, a decision was made to lay the groundwork for full integration of
+the SDN feature by elevating the management of networks and interfaces to a core
+component in the {pve} access control stack.
+In {pve} 8.1, two major milestones were achieved: firstly, DHCP integration was
+added to the IP address management (IPAM) feature, and secondly, the SDN
+integration is now installed by default.
+
+Current Status
+~~~~~~~~~~~~~~
+
+The current support status for the various layers of our SDN installation is as
+follows:
+
+- Core SDN, which includes VNet management and its integration with the {pve}
+ stack, is fully supported.
+- IPAM, including DHCP management for virtual guests, is in tech preview.
+- Complex routing via FRRouting and controller integration are in tech preview.
[[pvesdn_installation]]
Installation
------------
-To enable the experimental Software-Defined Network (SDN) integration, you need
-to install the `libpve-network-perl` package on every node:
+SDN Core
+~~~~~~~~
+
+Since {pve} 8.1 the core Software-Defined Network (SDN) packages are installed
+by default.
+
+If you upgrade from an older version, you need to install the
+`libpve-network-perl` package on every node:
----
apt update
apt install libpve-network-perl
----
-NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
+NOTE: {pve} version 7.0 and above have the `ifupdown2` package installed by
default. If you originally installed your system with an older version, you need
to explicitly install the `ifupdown2` package.
-After installation, you need to add the following line to the end of the
-`/etc/network/interfaces` configuration file, so that the SDN configuration gets
-included and activated.
+After installation, you need to ensure that the following line is present at the
+end of the `/etc/network/interfaces` configuration file on all nodes, so that
+the SDN configuration gets included and activated.
----
source /etc/network/interfaces.d/*
----
+[[pvesdn_install_dhcp_ipam]]
+DHCP IPAM
+~~~~~~~~~
-[[pvesdn_overview]]
-Overview
---------
+The DHCP integration into the built-in 'PVE' IP Address Management stack
+currently uses `dnsmasq` for giving out DHCP leases. This is currently opt-in.
-The {pve} SDN allows for separation and fine-grained control of virtual guest
-networks, using flexible, software-controlled configurations.
+To use that feature you need to install the `dnsmasq` package on every node:
-Separation is managed through *zones*, virtual networks (*VNets*), and
-*subnets*. A zone is its own virtually separated network area. A VNet is a
-virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
+----
+apt update
+apt install dnsmasq
+# disable default instance
+systemctl disable --now dnsmasq
+----
-Depending on the type of the zone, the network behaves differently and offers
-specific features, advantages, and limitations.
+[[pvesdn_install_frrouting]]
+FRRouting
+~~~~~~~~~
-Use cases for SDN range from an isolated private network on each individual node
-to complex overlay networks across multiple PVE clusters on different locations.
+The {pve} SDN stack uses the https://frrouting.org/[FRRouting] project for
+advanced setups. This is currently opt-in.
-After configuring an VNet in the cluster-wide datacenter SDN administration
-interface, it is available as a common Linux bridge, locally on each node, to be
-assigned to VMs and Containers.
+To use the SDN routing integration you need to install the `frr-pythontools`
+package on all nodes:
+----
+apt update
+apt install frr-pythontools
+----
[[pvesdn_main_configuration]]
-Main Configuration
-~~~~~~~~~~~~~~~~~~
+Configuration Overview
+----------------------
Configuration is done at the web UI at datacenter level, separated into the
following sections:
* xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
in complex setups
+* DHCP: Define a DHCP server for a zone that automatically allocates IPs for
+ guests in the IPAM and leases them to the guests via DHCP.
+
* xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
guests
* xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
- virtual guests' hostname and IP
- addresses
+ virtual guests' hostname and IP addresses
[[pvesdn_tech_and_config_overview]]
Technology & Configuration
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+--------------------------
The {pve} Software-Defined Network implementation uses standard Linux networking
as much as possible. The reason for this is that modern Linux networking
Tag:: The unique VLAN or VXLAN ID
VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
- in the quest.
+ in the guest.
[[pvesdn_config_subnet]]
-----------
Some zones implement a separated control and data plane that require an external
-external controller to manage the VNet's control plane.
+controller to manage the VNet's control plane.
Currently, only the `EVPN` zone requires an external controller.
The default built-in IPAM for your {pve} cluster.
+You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in
+the SDN section of the datacenter configuration. This UI can be used to create,
+update and delete IP mappings. This is particularly convenient in conjunction
+with the xref:pvesdn_config_dhcp[DHCP feature].
+
+If you are using DHCP, you can use the IPAM panel to create or edit leases for
+specific VMs, which enables you to change the IPs allocated via DHCP. When
+editing an IP of a VM that is using DHCP you must make sure to force the guest
+to acquire a new DHCP leases. This can usually be done by reloading the network
+stack of the guest or rebooting it.
[[pvesdn_ipam_plugin_netbox]]
NetBox IPAM Plugin
-~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~
link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
ttl:: The default TTL for records
+[[pvesdn_config_dhcp]]
+DHCP
+------
+
+The DHCP plugin in {pve} SDN can be used to automatically deploy a DHCP server
+for a Zone. It provides DHCP for all Subnets in a Zone that have a DHCP range
+configured. Currently the only available backend plugin for DHCP is the dnsmasq
+plugin.
+
+The DHCP plugin works by allocating an IP in the IPAM plugin configured in the
+Zone when adding a new network interface to a VM/CT. You can find more
+information on how to configure an IPAM in the
+xref:pvesdn_config_ipam[respective section of our documentation].
+
+When the VM starts, a mapping for the MAC address and IP gets created in the DHCP
+plugin of the zone. When the network interfaces is removed or the VM/CT are
+destroyed, then the entry in the IPAM and the DHCP server are deleted as well.
+
+NOTE: Some features (adding/editing/removing IP mappings) are currently only
+available when using the xref:pvesdn_ipam_plugin_pveipam[PVE IPAM plugin].
+
+
+Configuration
+~~~~~~~~~~~~~
+
+You can enable automatic DHCP for a zone in the Web UI via the Zones panel and
+enabling DHCP in the advanced options of a zone.
+
+NOTE: Currently only Simple Zones have support for automatic DHCP
+
+After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be
+configured for the subnets in a Zone. In order to that, go to the Vnets panel and
+select the Subnet for which you want to configure DHCP ranges. In the edit
+dialogue you can configure DHCP ranges in the respective Tab. Alternatively you
+can set DHCP ranges for a Subnet via the following CLI command:
+
+----
+pvesh set /cluster/sdn/vnets/<vnet>/subnets/<subnet>
+ -dhcp-range start-address=10.0.1.100,end-address=10.0.1.200
+ -dhcp-range start-address=10.0.2.100,end-address=10.0.2.200
+----
+
+You also need to have a gateway configured for the subnet - otherwise
+automatic DHCP will not work.
+
+The DHCP plugin will then allocate IPs in the IPAM only in the configured
+ranges.
+
+Do not forget to follow the installation steps for the
+xref:pvesdn_install_dhcp_ipam[dnsmasq DHCP plugin] as well.
+
+Plugins
+~~~~~~~
+
+Dnsmasq Plugin
+^^^^^^^^^^^^^^
+Currently this is the only DHCP plugin and therefore the plugin that gets used
+when you enable DHCP for a zone.
+
+.Installation
+For installation see the xref:pvesdn_install_dhcp_ipam[DHCP IPAM] section.
+
+.Configuration
+The plugin will create a new systemd service for each zone that dnsmasq gets
+deployed to. The name for the service is `dnsmasq@<zone>`. The lifecycle of this
+service is managed by the DHCP plugin.
+
+The plugin automatically generates the following configuration files in the
+folder `/etc/dnsmasq.d/<zone>`:
+
+`00-default.conf`::
+This contains the default global configuration for a dnsmasq instance.
+
+`10-<zone>-<subnet_cidr>.conf`::
+This file configures specific options for a subnet, such as the DNS server that
+should get configured via DHCP.
+
+`10-<zone>-<subnet_cidr>.ranges.conf`::
+This file configures the DHCP ranges for the dnsmasq instance.
+
+`ethers`::
+This file contains the MAC-address and IP mappings from the IPAM plugin. In
+order to override those mappings, please use the respective IPAM plugin rather
+than editing this file, as it will get overwritten by the dnsmasq plugin.
+
+You must not edit any of the above files, since they are managed by the DHCP
+plugin. In order to customize the dnsmasq configuration you can create
+additional files (e.g. `90-custom.conf`) in the configuration folder - they will
+not get changed by the dnsmasq DHCP plugin.
+
+Configuration files are read in order, so you can control the order of the
+configuration directives by naming your custom configuration files appropriately.
+
+DHCP leases are stored in the file `/var/lib/misc/dnsmasq.<zone>.leases`.
+
+When using the PVE IPAM plugin, you can update, create and delete DHCP leases.
+For more information please consult the documentation of
+xref:pvesdn_ipam_plugin_pveipam[the PVE IPAM plugin]. Changing DHCP leases is
+currently not supported for the other IPAM plugins.
+
[[pvesdn_setup_examples]]
Examples
--------
Simple Zone Example
~~~~~~~~~~~~~~~~~~~
-Simple zone networks create an isolated network for quests on a single host to
+Simple zone networks create an isolated network for guests on a single host to
connect to each other.
-TIP: connection between quests are possible if all quests reside on a same host
+TIP: connection between guests are possible if all guests reside on a same host
but cannot be reached on other nodes.
* Create a simple zone named `simple`.
* Add a VNet names `vnet1`.
* Create a Subnet with a gateway and the SNAT option enabled.
* This creates a network bridge `vnet1` on the node. Assign this bridge to the
- quests that shall join the network and configure an IP address.
+ guests that shall join the network and configure an IP address.
The network interface configuration in two VMs may look like this which allows
them to communicate via the 10.0.1.0/24 network.
Source NAT Example
~~~~~~~~~~~~~~~~~~
-If you want to allow outgoing connections for quests in the simple network zone
+If you want to allow outgoing connections for guests in the simple network zone
the simple zone offers a Source NAT (SNAT) option.
Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
SNAT: checked
----
-In the quests configure the static IP address inside the subnet's IP range.
+In the guests configure the static IP address inside the subnet's IP range.
The node itself will join this network with the Gateway IP '172.16.0.1' and
-function as the NAT gateway for quests within the subnet range.
+function as the NAT gateway for guests within the subnet range.
[[pvesdn_setup_example_vlan]]
Tag: 100
----
-Apply the configuration on the main SDN web-interface panel to create VNets
+Apply the configuration on the main SDN web interface panel to create VNets
locally on each node.
Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
Tag: 100000
----
-Apply the configuration on the main SDN web-interface panel to create VNets
+Apply the configuration on the main SDN web interface panel to create VNets
locally on each nodes.
Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
Gateway: 10.0.2.1
----
-Apply the configuration from the main SDN web-interface panel to create VNets
+Apply the configuration from the main SDN web interface panel to create VNets
locally on each node and generate the FRR configuration.
Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
projects / pve-docs.git / blobdiff