2 Software-Defined Network
3 ========================
8 The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
9 virtual networks (VNets) at the datacenter level.
11 WARNING: SDN is currently an **experimental feature** in {pve}. This
12 documentation for it is also still under development. Ask on our
13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
16 [[pvesdn_installation]]
20 To enable the experimental Software-Defined Network (SDN) integration, you need
21 to install the `libpve-network-perl` package on every node:
25 apt install libpve-network-perl
28 NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
29 default. If you originally installed your system with an older version, you need
30 to explicitly install the `ifupdown2` package.
32 After installation, you need to add the following line to the end of the
33 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
34 included and activated.
37 source /etc/network/interfaces.d/*
45 The {pve} SDN allows for separation and fine-grained control of virtual guest
46 networks, using flexible, software-controlled configurations.
48 Separation is managed through *zones*, virtual networks (*VNets*), and
49 *subnets*. A zone is its own virtually separated network area. A VNet is a
50 virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
52 Depending on the type of the zone, the network behaves differently and offers
53 specific features, advantages, and limitations.
55 Use cases for SDN range from an isolated private network on each individual node
56 to complex overlay networks across multiple PVE clusters on different locations.
58 After configuring an VNet in the cluster-wide datacenter SDN administration
59 interface, it is available as a common Linux bridge, locally on each node, to be
60 assigned to VMs and Containers.
63 [[pvesdn_main_configuration]]
67 Configuration is done at the web UI at datacenter level and is saved in files
68 located in the shared configuration file system at `/etc/pve/sdn`.
70 On the web interface, SDN features the following sections:
72 * SDN:: Here you get an overview of the current active SDN state, and you can
73 apply all pending changes to the whole cluster.
75 * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
78 * xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and
81 The Options category allows adding and managing additional services to be used
84 * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
87 * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
90 * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
91 virtual guests' hostname and IP
94 [[pvesdn_config_main_sdn]]
98 This is the main status panel. Here you can see the deployment status of zones
101 Pressing the 'Apply' button reloads the local configuration on all cluster
104 [[pvesdn_config_zone]]
108 A zone defines a virtually separated network. Zones are restricted to
109 specific nodes and assigned permissions, in order to restrict users to a certain
110 zone and its contained VNets.
112 Different technologies can be used for separation:
114 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
116 * VLAN: Virtual LANs are the classic method of subdividing a LAN
118 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
120 * VXLAN: Layer 2 VXLAN network via a UDP tunnel
122 * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
125 [[pvesdn_config_common_options]]
129 The following options are available for all zone types:
131 Nodes:: The nodes which the zone and associated VNets should be deployed on.
133 IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
134 zone. Optional, defaults to `pve`.
136 DNS:: DNS API server. Optional.
138 ReverseDNS:: Reverse DNS API server. Optional.
140 DNSZone:: DNS domain name. Used to register hostnames, such as
141 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
144 [[pvesdn_zone_plugin_simple]]
148 This is the simplest plugin. It will create an isolated VNet bridge. This
149 bridge is not linked to a physical interface, and VM traffic is only local on
151 It can be used in NAT or routed setups.
154 [[pvesdn_zone_plugin_vlan]]
158 The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
159 node's physical interface. It uses VLAN tagging defined in the VNet to isolate
160 the network segments. This allows connectivity of VMs between different nodes.
162 VLAN zone configuration options:
164 Bridge:: The local bridge or OVS switch, already configured on *each* node that
165 allows node-to-node connection.
168 [[pvesdn_zone_plugin_qinq]]
172 QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
173 isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
174 whereas the inner VLAN tag is defined by the VNet.
176 NOTE: Your physical network switches must support stacked VLANs for this
179 QinQ zone configuration options:
181 Bridge:: A local, VLAN-aware bridge that is already configured on each local
184 Service VLAN:: The main VLAN tag of this zone
186 Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
187 802.1ad service VLAN type.
189 MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
190 For example, you must reduce the MTU to `1496` if you physical interface MTU is
194 [[pvesdn_zone_plugin_vxlan]]
198 The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
199 (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
200 datagrams using the default destination port `4789`.
202 You have to configure the underlay network yourself to enable UDP connectivity
205 You can, for example, create a VXLAN overlay network on top of public internet,
206 appearing to the VMs as if they share the same local Layer 2 network.
208 WARNING: VXLAN on its own does does not provide any encryption. When joining
209 multiple sites via VXLAN, make sure to establish a secure connection between
210 the site, for example by using a site-to-site VPN.
212 VXLAN zone configuration options:
214 Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
215 can be external nodes reachable at this IP address.
216 All nodes in the cluster need to be mentioned here.
218 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
219 lower than the outgoing physical interface.
222 [[pvesdn_zone_plugin_evpn]]
226 The EVPN zone creates a routable Layer 3 network, capable of spanning across
227 multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
228 the routing protocol.
230 The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
231 IP is the same on each node, meaning a virtual guest can use this address as
234 Routing can work across VNets from different zones through a VRF (Virtual
235 Routing and Forwarding) interface.
237 EVPN zone configuration options:
239 VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
240 It must be different than the VXLAN-ID of the VNets.
242 Controller:: The EVPN-controller to use for this zone. (See controller plugins
245 VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
246 zone. Will be auto-generated if not defined.
248 Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
249 network, through the real network. The configured nodes will announce a
250 default route in the EVPN network. Optional.
252 Primary Exit Node:: If you use multiple exit nodes, force traffic through this
253 primary exit node, instead of load-balancing on all nodes. Optional but
254 necessary if you want to use SNAT or if your upstream router doesn't support
257 Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
258 service from an exit node. (By default, the exit nodes only allow forwarding
259 traffic between real network and EVPN network). Optional.
261 Advertise Subnets:: Announce the full subnet in the EVPN network.
262 If you have silent VMs/CTs (for example, if you have multiple IPs and the
263 anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
264 able to be reached inside the EVPN network). Optional.
266 Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
267 packets. This is required if you use floating IPs in your VMs (IP and MAC
268 addresses are being moved between systems). Optional.
270 Route-target Import:: Allows you to import a list of external EVPN route
271 targets. Used for cross-DC or different EVPN network interconnects. Optional.
273 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
274 less than the maximal MTU of the outgoing physical interface. Optional,
278 [[pvesdn_config_vnets]]
282 After creating a virtual network (VNet) through the SDN GUI, a local network
283 interface with the same name is available on each node. To connect a guest to the
284 VNet, assign the interface to the guest and set the IP address accordingly.
286 Depending on the zone, these options have different meanings and are explained
287 in the respective zone section in this document.
289 WARNING: In the current state, some options may have no effect or won't work in
292 VNet configuration options:
294 ID:: An up to 8 character ID to identify a VNet
296 Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
298 Zone:: The associated zone for this VNet
300 Tag:: The unique VLAN or VXLAN ID
302 VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
306 [[pvesdn_config_subnet]]
310 A subnet define a specific IP range, described by the CIDR network address.
311 Each VNet, can have one or more subnets.
313 A subnet can be used to:
315 * Restrict the IP addresses you can define on a specific VNet
316 * Assign routes/gateways on a VNet in layer 3 zones
317 * Enable SNAT on a VNet in layer 3 zones
318 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
319 * DNS registration through DNS plugins
321 If an IPAM server is associated with the subnet zone, the subnet prefix will be
322 automatically registered in the IPAM.
324 Subnet configuration options:
326 ID:: A CIDR network address, for example 10.0.0.0/8
328 Gateway:: The IP address of the network's default gateway. On layer 3 zones
329 (Simple/EVPN plugins), it will be deployed on the VNet.
331 SNAT:: Enable Source NAT which allows VMs from inside a
332 VNet to connect to the outside network by forwarding the packets to the nodes
333 outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
336 DNS Zone Prefix:: Add a prefix to the domain registration, like
337 <hostname>.prefix.<domain> Optional.
340 [[pvesdn_config_controllers]]
344 Some zones implement a separated control and data plane that require an external
345 external controller to manage the VNet's control plane.
347 Currently, only the `EVPN` zone requires an external controller.
350 [[pvesdn_controller_plugin_evpn]]
354 The `EVPN`, zone requires an external controller to manage the control plane.
355 The EVPN controller plugin configures the Free Range Routing (frr) router.
357 To enable the EVPN controller, you need to install frr on every node that shall
358 participate in the EVPN zone.
361 apt install frr frr-pythontools
364 EVPN controller configuration options:
366 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
367 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
368 breaking global routing by mistake.
370 Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
371 external nodes or route reflector servers)
374 [[pvesdn_controller_plugin_BGP]]
378 The BGP controller is not used directly by a zone.
379 You can use it to configure FRR to manage BGP peers.
381 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
382 It can also be used to export EVPN routes to an external BGP peer.
384 NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
387 BGP controller configuration options:
389 Node:: The node of this BGP controller
391 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
392 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
393 you could break global routing by mistake.
395 Peer:: A list of peer IP addresses you want to communicate with using the
396 underlying BGP network.
398 EBGP:: If your peer's remote-AS is different, this enables EBGP.
400 Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
403 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
404 not directly connected or they use loopback.
406 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
409 [[pvesdn_controller_plugin_ISIS]]
413 The ISIS controller is not used directly by a zone.
414 You can use it to configure FRR to export EVPN routes to an ISIS domain.
416 ISIS controller configuration options:
418 Node:: The node of this ISIS controller.
420 Domain:: A unique ISIS domain.
422 Network Entity Title:: A Unique ISIS network address that identifies this node.
424 Interfaces:: A list of physical interface(s) used by ISIS.
426 Loopback:: Use a loopback or dummy interface as the source of the EVPN network
430 [[pvesdn_config_ipam]]
434 IP Address Management (IPAM) tools manage the IP addresses of clients on the
435 network. SDN in {pve} uses IPAM for example to find free IP addresses for new
438 A single IPAM instance can be associated with one or more zones.
441 [[pvesdn_ipam_plugin_pveipam]]
445 The default built-in IPAM for your {pve} cluster.
448 [[pvesdn_ipam_plugin_netbox]]
452 link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
453 Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
455 To integrate NetBox with {pve} SDN, create an API token in NetBox as described
456 here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
458 The NetBox configuration properties are:
460 URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
462 Token:: An API access token
465 [[pvesdn_ipam_plugin_phpipam]]
469 In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
470 an API token with admin privileges to the application.
472 The phpIPAM configuration properties are:
474 URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
476 Token:: An API access token
478 Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
479 installations use `sectionid=1` for customers.
482 [[pvesdn_config_dns]]
486 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
487 of your hostname and IP address. A DNS configuration is associated with one or
488 more zones, to provide DNS registration for all the subnet IPs configured for
491 [[pvesdn_dns_plugin_powerdns]]
494 https://doc.powerdns.com/authoritative/http-api/index.html
496 You need to enable the web server and the API in your PowerDNS config:
500 api-key=arandomgeneratedstring
505 The PowerDNS configuration options are:
507 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
509 key:: An API access key
511 ttl:: The default TTL for records
514 [[pvesdn_setup_examples]]
518 This section presents multiple configuration examples tailored for common SDN
519 use cases. It aims to offer tangible implementations, providing additional
520 details to enhance comprehension of the available configuration options.
523 [[pvesdn_setup_example_simple]]
527 Simple zone networks create an isolated network for quests on a single host to
528 connect to each other.
530 TIP: connection between quests are possible if all quests reside on a same host
531 but cannot be reached on other nodes.
533 * Create a simple zone named `simple`.
534 * Add a VNet names `vnet1`.
535 * Create a Subnet with a gateway and the SNAT option enabled.
536 * This creates a network bridge `vnet1` on the node. Assign this bridge to the
537 quests that shall join the network and configure an IP address.
539 The network interface configuration in two VMs may look like this which allows
540 them to communicate via the 10.0.1.0/24 network.
544 iface ens19 inet static
550 iface ens19 inet static
555 [[pvesdn_setup_example_nat]]
559 If you want to allow outgoing connections for quests in the simple network zone
560 the simple zone offers a Source NAT (SNAT) option.
562 Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
563 Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
566 Subnet: 172.16.0.0/24
571 In the quests configure the static IP address inside the subnet's IP range.
573 The node itself will join this network with the Gateway IP '172.16.0.1' and
574 function as the NAT gateway for quests within the subnet range.
577 [[pvesdn_setup_example_vlan]]
581 When VMs on different nodes need to communicate through an isolated network, the
582 VLAN zone allows network level isolation using VLAN tags.
584 Create a VLAN zone named `myvlanzone`:
591 Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
600 Apply the configuration through the main SDN panel, to create VNets locally on
603 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
605 Use the following network configuration for this VM:
609 iface eth0 inet static
610 address 10.0.3.100/24
613 Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
616 Use the following network configuration for this VM:
620 iface eth0 inet static
621 address 10.0.3.101/24
624 Following this, you should be able to ping between both VMs using that network.
627 [[pvesdn_setup_example_qinq]]
632 This example configures two QinQ zones and adds two VMs to each zone to
633 demonstrate the additional layer of VLAN tags which allows the configuration of
636 A typical use case for this configuration is a hosting provider that provides an
637 isolated network to customers for VM communication but isolates the VMs from
640 Create a QinQ zone named `qinqzone1` with service VLAN 20
648 Create another QinQ zone named `qinqzone2` with service VLAN 30
655 Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
664 Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
672 Apply the configuration on the main SDN web-interface panel to create VNets
673 locally on each node.
675 Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
676 interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
679 Inside the VM, configure the IP addresses of the interfaces, for example via
680 `/etc/network/interfaces`:
684 iface eth0 inet static
685 address 10.0.3.101/24
687 // TODO: systemd-network example
688 Configure all four VMs to have IP addresses from the '10.0.3.101' to
691 Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
692 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
693 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
696 [[pvesdn_setup_example_vxlan]]
700 The example assumes a cluster with three nodes, with the node IP addresses
701 192.168.0.1, 192.168.0.2 and 192.168.0.3.
703 Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
704 peer address list. Use the default MTU of 1450 or configure accordingly.
708 Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
711 Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
720 Apply the configuration on the main SDN web-interface panel to create VNets
721 locally on each nodes.
723 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
725 Use the following network configuration for this VM (note the lower MTU).
729 iface eth0 inet static
730 address 10.0.3.100/24
734 Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
737 Use the following network configuration for this VM:
741 iface eth0 inet static
742 address 10.0.3.101/24
746 Then, you should be able to ping between between 'vm1' and 'vm2'.
749 [[pvesdn_setup_example_evpn]]
753 The example assumes a cluster with three nodes (node1, node2, node3) with IP
754 addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
756 Create an EVPN controller, using a private ASN number and the above node
762 Peers: 192.168.0.1,192.168.0.2,192.168.0.3
765 Create an EVPN zone named `myevpnzone`, assign the previously created
766 EVPN-controller and define 'node1' and 'node2' as exit nodes.
771 Controller: myevpnctl
773 VNet MAC Address: 32:F4:05:FE:6C:0A
774 Exit Nodes: node1,node2
777 Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
785 Create a subnet on `myvnet1`:
792 Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
800 Create a different subnet on `myvnet2``:
807 Apply the configuration from the main SDN web-interface panel to create VNets
808 locally on each node and generate the FRR configuration.
810 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
812 Use the following network configuration for 'vm1':
816 iface eth0 inet static
817 address 10.0.1.100/24
822 Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
825 Use the following network configuration for 'vm2':
829 iface eth0 inet static
830 address 10.0.2.100/24
836 Now you should be able to ping vm2 from vm1, and vm1 from vm2.
838 If you ping an external IP from 'vm2' on the non-gateway node3, the packet
839 will go to the configured 'myvnet2' gateway, then will be routed to the exit
840 nodes ('node1' or 'node2') and from there it will leave those nodes over the
841 default gateway configured on node1 or node2.
843 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
844 networks to node1 and node2 on your external gateway, so that the public network
847 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
848 and 10.0.2.0/24 in this example), will be announced dynamically.
855 Multiple EVPN Exit Nodes
856 ~~~~~~~~~~~~~~~~~~~~~~~~
858 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
859 Reverse Path Filter) option, because packets can arrive at one node but go out
862 Add the following to `/etc/sysctl.conf`:
865 net.ipv4.conf.default.rp_filter=0
866 net.ipv4.conf.all.rp_filter=0
869 VXLAN IPSEC Encryption
870 ~~~~~~~~~~~~~~~~~~~~~~
872 To add IPSEC encryption on top of a VXLAN, this example shows how to use
875 You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
876 IPv6 to handle encryption.
878 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
879 + 50 (VXLAN) == 1500).
881 Install strongswan on the host.
884 apt install strongswan
887 Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
888 the VXLAN UDP port '4789'.
892 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
894 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
897 rightsubnet=%dynamic[udp/4789]
904 leftsubnet=%dynamic[udp/4789]
910 Generate a pre-shared key with:
913 openssl rand -base64 128
916 and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
919 : PSK <generatedbase64key>
922 Copy the PSK and the configuration to all nodes participating in the VXLAN network.