2 Software-Defined Network
3 ========================
8 The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
9 virtual networks (VNets) at the datacenter level.
11 WARNING: SDN is currently an **experimental feature** in {pve}. This
12 documentation for it is also still under development. Ask on our
13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
16 [[pvesdn_installation]]
20 To enable the experimental Software-Defined Network (SDN) integration, you need
21 to install the `libpve-network-perl` package on every node:
25 apt install libpve-network-perl
28 NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
29 default. If you originally installed your system with an older version, you need
30 to explicitly install the `ifupdown2` package.
32 After installation, you need to add the following line to the end of the
33 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
34 included and activated.
37 source /etc/network/interfaces.d/*
45 The {pve} SDN allows for separation and fine-grained control of virtual guest
46 networks, using flexible, software-controlled configurations.
48 Separation is managed through *zones*, virtual networks (*VNets*), and
49 *subnets*. A zone is its own virtually separated network area. A VNet is a
50 virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
52 Depending on the type of the zone, the network behaves differently and offers
53 specific features, advantages, and limitations.
55 Use cases for SDN range from an isolated private network on each individual node
56 to complex overlay networks across multiple PVE clusters on different locations.
58 After configuring an VNet in the cluster-wide datacenter SDN administration
59 interface, it is available as a common Linux bridge, locally on each node, to be
60 assigned to VMs and Containers.
63 [[pvesdn_main_configuration]]
67 Configuration is done at the web UI at datacenter level, separated into the
70 * SDN:: Here you get an overview of the current active SDN state, and you can
71 apply all pending changes to the whole cluster.
73 * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
76 * xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and
79 The Options category allows adding and managing additional services to be used
82 * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
85 * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
88 * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
89 virtual guests' hostname and IP
92 [[pvesdn_tech_and_config_overview]]
93 Technology & Configuration
94 ~~~~~~~~~~~~~~~~~~~~~~~~~~
96 The {pve} Software-Defined Network implementation uses standard Linux networking
97 as much as possible. The reason for this is that modern Linux networking
98 provides almost all needs for a feature full SDN implementation and avoids adding
99 external dependencies and reduces the overall amount of components that can
102 The {pve} SDN configurations are located in `/etc/pve/sdn`, which is shared with
103 all other cluster nodes through the {pve} xref:chapter_pmxcfs[configuration file system].
104 Those configurations get translated to the respective configuration formats of
105 the tools that manage the underlying network stack (for example `ifupdown2` or
108 New changes are not immediately applied but recorded as pending first. You can
109 then apply a set of different changes all at once in the main 'SDN' overview
110 panel on the web interface. This system allows to roll-out various changes as
113 The SDN tracks the rolled-out state through the '.running-config' and '.version'
114 files located in '/etc/pve/sdn'.
116 // TODO: extend implementation and technology details.
118 [[pvesdn_config_zone]]
122 A zone defines a virtually separated network. Zones are restricted to
123 specific nodes and assigned permissions, in order to restrict users to a certain
124 zone and its contained VNets.
126 Different technologies can be used for separation:
128 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
130 * VLAN: Virtual LANs are the classic method of subdividing a LAN
132 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
134 * VXLAN: Layer 2 VXLAN network via a UDP tunnel
136 * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
139 [[pvesdn_config_common_options]]
143 The following options are available for all zone types:
145 Nodes:: The nodes which the zone and associated VNets should be deployed on.
147 IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
148 zone. Optional, defaults to `pve`.
150 DNS:: DNS API server. Optional.
152 ReverseDNS:: Reverse DNS API server. Optional.
154 DNSZone:: DNS domain name. Used to register hostnames, such as
155 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
158 [[pvesdn_zone_plugin_simple]]
162 This is the simplest plugin. It will create an isolated VNet bridge. This
163 bridge is not linked to a physical interface, and VM traffic is only local on
165 It can be used in NAT or routed setups.
168 [[pvesdn_zone_plugin_vlan]]
172 The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
173 node's physical interface. It uses VLAN tagging defined in the VNet to isolate
174 the network segments. This allows connectivity of VMs between different nodes.
176 VLAN zone configuration options:
178 Bridge:: The local bridge or OVS switch, already configured on *each* node that
179 allows node-to-node connection.
182 [[pvesdn_zone_plugin_qinq]]
186 QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
187 isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
188 whereas the inner VLAN tag is defined by the VNet.
190 NOTE: Your physical network switches must support stacked VLANs for this
193 QinQ zone configuration options:
195 Bridge:: A local, VLAN-aware bridge that is already configured on each local
198 Service VLAN:: The main VLAN tag of this zone
200 Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
201 802.1ad service VLAN type.
203 MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
204 For example, you must reduce the MTU to `1496` if you physical interface MTU is
208 [[pvesdn_zone_plugin_vxlan]]
212 The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
213 (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
214 datagrams using the default destination port `4789`.
216 You have to configure the underlay network yourself to enable UDP connectivity
219 You can, for example, create a VXLAN overlay network on top of public internet,
220 appearing to the VMs as if they share the same local Layer 2 network.
222 WARNING: VXLAN on its own does does not provide any encryption. When joining
223 multiple sites via VXLAN, make sure to establish a secure connection between
224 the site, for example by using a site-to-site VPN.
226 VXLAN zone configuration options:
228 Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
229 can be external nodes reachable at this IP address.
230 All nodes in the cluster need to be mentioned here.
232 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
233 lower than the outgoing physical interface.
236 [[pvesdn_zone_plugin_evpn]]
240 The EVPN zone creates a routable Layer 3 network, capable of spanning across
241 multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
242 the routing protocol.
244 The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
245 IP is the same on each node, meaning a virtual guest can use this address as
248 Routing can work across VNets from different zones through a VRF (Virtual
249 Routing and Forwarding) interface.
251 EVPN zone configuration options:
253 VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
254 It must be different than the VXLAN-ID of the VNets.
256 Controller:: The EVPN-controller to use for this zone. (See controller plugins
259 VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
260 zone. Will be auto-generated if not defined.
262 Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
263 network, through the real network. The configured nodes will announce a
264 default route in the EVPN network. Optional.
266 Primary Exit Node:: If you use multiple exit nodes, force traffic through this
267 primary exit node, instead of load-balancing on all nodes. Optional but
268 necessary if you want to use SNAT or if your upstream router doesn't support
271 Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
272 service from an exit node. (By default, the exit nodes only allow forwarding
273 traffic between real network and EVPN network). Optional.
275 Advertise Subnets:: Announce the full subnet in the EVPN network.
276 If you have silent VMs/CTs (for example, if you have multiple IPs and the
277 anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
278 able to be reached inside the EVPN network). Optional.
280 Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
281 packets. This is required if you use floating IPs in your VMs (IP and MAC
282 addresses are being moved between systems). Optional.
284 Route-target Import:: Allows you to import a list of external EVPN route
285 targets. Used for cross-DC or different EVPN network interconnects. Optional.
287 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
288 less than the maximal MTU of the outgoing physical interface. Optional,
292 [[pvesdn_config_vnets]]
296 After creating a virtual network (VNet) through the SDN GUI, a local network
297 interface with the same name is available on each node. To connect a guest to the
298 VNet, assign the interface to the guest and set the IP address accordingly.
300 Depending on the zone, these options have different meanings and are explained
301 in the respective zone section in this document.
303 WARNING: In the current state, some options may have no effect or won't work in
306 VNet configuration options:
308 ID:: An up to 8 character ID to identify a VNet
310 Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
312 Zone:: The associated zone for this VNet
314 Tag:: The unique VLAN or VXLAN ID
316 VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
320 [[pvesdn_config_subnet]]
324 A subnet define a specific IP range, described by the CIDR network address.
325 Each VNet, can have one or more subnets.
327 A subnet can be used to:
329 * Restrict the IP addresses you can define on a specific VNet
330 * Assign routes/gateways on a VNet in layer 3 zones
331 * Enable SNAT on a VNet in layer 3 zones
332 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
333 * DNS registration through DNS plugins
335 If an IPAM server is associated with the subnet zone, the subnet prefix will be
336 automatically registered in the IPAM.
338 Subnet configuration options:
340 ID:: A CIDR network address, for example 10.0.0.0/8
342 Gateway:: The IP address of the network's default gateway. On layer 3 zones
343 (Simple/EVPN plugins), it will be deployed on the VNet.
345 SNAT:: Enable Source NAT which allows VMs from inside a
346 VNet to connect to the outside network by forwarding the packets to the nodes
347 outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
350 DNS Zone Prefix:: Add a prefix to the domain registration, like
351 <hostname>.prefix.<domain> Optional.
354 [[pvesdn_config_controllers]]
358 Some zones implement a separated control and data plane that require an external
359 external controller to manage the VNet's control plane.
361 Currently, only the `EVPN` zone requires an external controller.
364 [[pvesdn_controller_plugin_evpn]]
368 The `EVPN`, zone requires an external controller to manage the control plane.
369 The EVPN controller plugin configures the Free Range Routing (frr) router.
371 To enable the EVPN controller, you need to install frr on every node that shall
372 participate in the EVPN zone.
375 apt install frr frr-pythontools
378 EVPN controller configuration options:
380 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
381 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
382 breaking global routing by mistake.
384 Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
385 external nodes or route reflector servers)
388 [[pvesdn_controller_plugin_BGP]]
392 The BGP controller is not used directly by a zone.
393 You can use it to configure FRR to manage BGP peers.
395 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
396 It can also be used to export EVPN routes to an external BGP peer.
398 NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
401 BGP controller configuration options:
403 Node:: The node of this BGP controller
405 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
406 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
407 you could break global routing by mistake.
409 Peer:: A list of peer IP addresses you want to communicate with using the
410 underlying BGP network.
412 EBGP:: If your peer's remote-AS is different, this enables EBGP.
414 Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
417 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
418 not directly connected or they use loopback.
420 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
423 [[pvesdn_controller_plugin_ISIS]]
427 The ISIS controller is not used directly by a zone.
428 You can use it to configure FRR to export EVPN routes to an ISIS domain.
430 ISIS controller configuration options:
432 Node:: The node of this ISIS controller.
434 Domain:: A unique ISIS domain.
436 Network Entity Title:: A Unique ISIS network address that identifies this node.
438 Interfaces:: A list of physical interface(s) used by ISIS.
440 Loopback:: Use a loopback or dummy interface as the source of the EVPN network
444 [[pvesdn_config_ipam]]
448 IP Address Management (IPAM) tools manage the IP addresses of clients on the
449 network. SDN in {pve} uses IPAM for example to find free IP addresses for new
452 A single IPAM instance can be associated with one or more zones.
455 [[pvesdn_ipam_plugin_pveipam]]
459 The default built-in IPAM for your {pve} cluster.
462 [[pvesdn_ipam_plugin_netbox]]
466 link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
467 Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
469 To integrate NetBox with {pve} SDN, create an API token in NetBox as described
470 here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
472 The NetBox configuration properties are:
474 URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
476 Token:: An API access token
479 [[pvesdn_ipam_plugin_phpipam]]
483 In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
484 an API token with admin privileges to the application.
486 The phpIPAM configuration properties are:
488 URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
490 Token:: An API access token
492 Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
493 installations use `sectionid=1` for customers.
496 [[pvesdn_config_dns]]
500 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
501 of your hostname and IP address. A DNS configuration is associated with one or
502 more zones, to provide DNS registration for all the subnet IPs configured for
505 [[pvesdn_dns_plugin_powerdns]]
508 https://doc.powerdns.com/authoritative/http-api/index.html
510 You need to enable the web server and the API in your PowerDNS config:
514 api-key=arandomgeneratedstring
519 The PowerDNS configuration options are:
521 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
523 key:: An API access key
525 ttl:: The default TTL for records
528 [[pvesdn_setup_examples]]
532 This section presents multiple configuration examples tailored for common SDN
533 use cases. It aims to offer tangible implementations, providing additional
534 details to enhance comprehension of the available configuration options.
537 [[pvesdn_setup_example_simple]]
541 Simple zone networks create an isolated network for quests on a single host to
542 connect to each other.
544 TIP: connection between quests are possible if all quests reside on a same host
545 but cannot be reached on other nodes.
547 * Create a simple zone named `simple`.
548 * Add a VNet names `vnet1`.
549 * Create a Subnet with a gateway and the SNAT option enabled.
550 * This creates a network bridge `vnet1` on the node. Assign this bridge to the
551 quests that shall join the network and configure an IP address.
553 The network interface configuration in two VMs may look like this which allows
554 them to communicate via the 10.0.1.0/24 network.
558 iface ens19 inet static
564 iface ens19 inet static
569 [[pvesdn_setup_example_nat]]
573 If you want to allow outgoing connections for quests in the simple network zone
574 the simple zone offers a Source NAT (SNAT) option.
576 Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
577 Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
580 Subnet: 172.16.0.0/24
585 In the quests configure the static IP address inside the subnet's IP range.
587 The node itself will join this network with the Gateway IP '172.16.0.1' and
588 function as the NAT gateway for quests within the subnet range.
591 [[pvesdn_setup_example_vlan]]
595 When VMs on different nodes need to communicate through an isolated network, the
596 VLAN zone allows network level isolation using VLAN tags.
598 Create a VLAN zone named `myvlanzone`:
605 Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
614 Apply the configuration through the main SDN panel, to create VNets locally on
617 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
619 Use the following network configuration for this VM:
623 iface eth0 inet static
624 address 10.0.3.100/24
627 Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
630 Use the following network configuration for this VM:
634 iface eth0 inet static
635 address 10.0.3.101/24
638 Following this, you should be able to ping between both VMs using that network.
641 [[pvesdn_setup_example_qinq]]
646 This example configures two QinQ zones and adds two VMs to each zone to
647 demonstrate the additional layer of VLAN tags which allows the configuration of
650 A typical use case for this configuration is a hosting provider that provides an
651 isolated network to customers for VM communication but isolates the VMs from
654 Create a QinQ zone named `qinqzone1` with service VLAN 20
662 Create another QinQ zone named `qinqzone2` with service VLAN 30
669 Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
678 Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
686 Apply the configuration on the main SDN web-interface panel to create VNets
687 locally on each node.
689 Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
690 interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
693 Inside the VM, configure the IP addresses of the interfaces, for example via
694 `/etc/network/interfaces`:
698 iface eth0 inet static
699 address 10.0.3.101/24
701 // TODO: systemd-network example
702 Configure all four VMs to have IP addresses from the '10.0.3.101' to
705 Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
706 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
707 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
710 [[pvesdn_setup_example_vxlan]]
714 The example assumes a cluster with three nodes, with the node IP addresses
715 192.168.0.1, 192.168.0.2 and 192.168.0.3.
717 Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
718 peer address list. Use the default MTU of 1450 or configure accordingly.
722 Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
725 Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
734 Apply the configuration on the main SDN web-interface panel to create VNets
735 locally on each nodes.
737 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
739 Use the following network configuration for this VM (note the lower MTU).
743 iface eth0 inet static
744 address 10.0.3.100/24
748 Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
751 Use the following network configuration for this VM:
755 iface eth0 inet static
756 address 10.0.3.101/24
760 Then, you should be able to ping between between 'vm1' and 'vm2'.
763 [[pvesdn_setup_example_evpn]]
767 The example assumes a cluster with three nodes (node1, node2, node3) with IP
768 addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
770 Create an EVPN controller, using a private ASN number and the above node
776 Peers: 192.168.0.1,192.168.0.2,192.168.0.3
779 Create an EVPN zone named `myevpnzone`, assign the previously created
780 EVPN-controller and define 'node1' and 'node2' as exit nodes.
785 Controller: myevpnctl
787 VNet MAC Address: 32:F4:05:FE:6C:0A
788 Exit Nodes: node1,node2
791 Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
799 Create a subnet on `myvnet1`:
806 Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
814 Create a different subnet on `myvnet2``:
821 Apply the configuration from the main SDN web-interface panel to create VNets
822 locally on each node and generate the FRR configuration.
824 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
826 Use the following network configuration for 'vm1':
830 iface eth0 inet static
831 address 10.0.1.100/24
836 Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
839 Use the following network configuration for 'vm2':
843 iface eth0 inet static
844 address 10.0.2.100/24
850 Now you should be able to ping vm2 from vm1, and vm1 from vm2.
852 If you ping an external IP from 'vm2' on the non-gateway node3, the packet
853 will go to the configured 'myvnet2' gateway, then will be routed to the exit
854 nodes ('node1' or 'node2') and from there it will leave those nodes over the
855 default gateway configured on node1 or node2.
857 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
858 networks to node1 and node2 on your external gateway, so that the public network
861 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
862 and 10.0.2.0/24 in this example), will be announced dynamically.
869 Multiple EVPN Exit Nodes
870 ~~~~~~~~~~~~~~~~~~~~~~~~
872 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
873 Reverse Path Filter) option, because packets can arrive at one node but go out
876 Add the following to `/etc/sysctl.conf`:
879 net.ipv4.conf.default.rp_filter=0
880 net.ipv4.conf.all.rp_filter=0
883 VXLAN IPSEC Encryption
884 ~~~~~~~~~~~~~~~~~~~~~~
886 To add IPSEC encryption on top of a VXLAN, this example shows how to use
889 You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
890 IPv6 to handle encryption.
892 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
893 + 50 (VXLAN) == 1500).
895 Install strongswan on the host.
898 apt install strongswan
901 Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
902 the VXLAN UDP port '4789'.
906 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
908 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
911 rightsubnet=%dynamic[udp/4789]
918 leftsubnet=%dynamic[udp/4789]
924 Generate a pre-shared key with:
927 openssl rand -base64 128
930 and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
933 : PSK <generatedbase64key>
936 Copy the PSK and the configuration to all nodes participating in the VXLAN network.