2 Software Defined Network
3 ========================
8 The **S**oftware **D**efined **N**etwork (SDN) feature allows you to create
9 virtual networks (VNets) at the datacenter level.
11 WARNING: SDN is currently an **experimental feature** in {pve}. This
12 documentation for it is also still under development. Ask on our
13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
16 [[pvesdn_installation]]
20 To enable the experimental Software Defined Network (SDN) integration, you need
21 to install the `libpve-network-perl` and `ifupdown2` packages on every node:
25 apt install libpve-network-perl ifupdown2
28 NOTE: {pve} version 7 and above come installed with ifupdown2.
30 After this, you need to add the following line to the end of the
31 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
32 included and activated.
35 source /etc/network/interfaces.d/*
42 The {pve} SDN allows for separation and fine-grained control of virtual guest
43 networks, using flexible, software-controlled configurations.
45 Separation is managed through zones, where a zone is its own virtual separated
46 network area. A 'VNet' is a type of a virtual network connected to a zone.
47 Depending on which type or plugin the zone uses, it can behave differently and
48 offer different features, advantages, and disadvantages. Normally, a 'VNet'
49 appears as a common Linux bridge with either a VLAN or 'VXLAN' tag, however,
50 some can also use layer 3 routing for control. 'VNets' are deployed locally on
51 each node, after being configured from the cluster-wide datacenter SDN
52 administration interface.
58 Configuration is done at the datacenter (cluster-wide) level and is saved in
59 files located in the shared configuration file system:
62 On the web-interface, SDN features 3 main sections:
64 * SDN: An overview of the SDN state
66 * Zones: Create and manage the virtually separated network zones
68 * VNets: Create virtual network bridges and manage subnets
70 In addition to this, the following options are offered:
72 * Controller: For controlling layer 3 routing in complex setups
74 * Subnets: Used to defined IP networks on VNets
76 * IPAM: Enables the use of external tools for IP address management (guest
79 * DNS: Define a DNS server API for registering virtual guests' hostname and IP
82 [[pvesdn_config_main_sdn]]
87 This is the main status panel. Here you can see the deployment status of zones
90 The 'Apply' button is used to push and reload local configuration on all cluster
94 [[pvesdn_local_deployment_monitoring]]
95 Local Deployment Monitoring
96 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
98 After applying the configuration through the main SDN panel,
99 the local network configuration is generated locally on each node in
100 the file `/etc/network/interfaces.d/sdn`, and reloaded with ifupdown2.
102 You can monitor the status of local zones and VNets through the main tree.
105 [[pvesdn_config_zone]]
109 A zone defines a virtually separated network. Zones can be restricted to
110 specific nodes and assigned permissions, in order to restrict users to a certain
111 zone and its contained VNets.
113 Different technologies can be used for separation:
115 * VLAN: Virtual LANs are the classic method of subdividing a LAN
117 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
119 * VXLAN: Layer2 VXLAN
121 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
123 * EVPN (BGP EVPN): VXLAN using layer 3 border gateway protocol (BGP) routing
128 The following options are available for all zone types:
130 nodes:: The nodes which the zone and associated VNets should be deployed on
132 ipam:: Optional. Use an IP Address Management (IPAM) tool to manage IPs in the
135 dns:: Optional. DNS API server.
137 reversedns:: Optional. Reverse DNS API server.
139 dnszone:: Optional. DNS domain name. Used to register hostnames, such as
140 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server.
143 [[pvesdn_zone_plugin_simple]]
147 This is the simplest plugin. It will create an isolated VNet bridge.
148 This bridge is not linked to a physical interface, and VM traffic is only
149 local between the node(s).
150 It can also be used in NAT or routed setups.
152 [[pvesdn_zone_plugin_vlan]]
156 This plugin reuses an existing local Linux or OVS bridge, and manages the VLANs
157 on it. The benefit of using the SDN module is that you can create different
158 zones with specific VNet VLAN tags, and restrict virtual machines to separated
161 Specific `VLAN` configuration options:
163 bridge:: Reuse this local bridge or OVS switch, already configured on *each*
166 [[pvesdn_zone_plugin_qinq]]
170 QinQ also known as VLAN stacking, wherein the first VLAN tag is defined for the
171 zone (the 'service-vlan'), and the second VLAN tag is defined for the
174 NOTE: Your physical network switches must support stacked VLANs for this
177 Below are the configuration options specific to QinQ:
179 bridge:: A local, VLAN-aware bridge that is already configured on each local
182 service vlan:: The main VLAN tag of this zone
184 service vlan protocol:: Allows you to choose between an 802.1q (default) or
185 802.1ad service VLAN type.
187 mtu:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
188 For example, you must reduce the MTU to `1496` if you physical interface MTU is
191 [[pvesdn_zone_plugin_vxlan]]
195 The VXLAN plugin establishes a tunnel (overlay) on top of an existing
196 network (underlay). This encapsulates layer 2 Ethernet frames within layer
197 4 UDP datagrams, using `4789` as the default destination port. You can, for
198 example, create a private IPv4 VXLAN network on top of public internet network
201 This is a layer 2 tunnel only, so no routing between different VNets is
204 Each VNet will have a specific VXLAN ID in the range 1 - 16777215.
206 Specific EVPN configuration options:
208 peers address list:: A list of IP addresses from each node through which you
209 want to communicate. Can also be external nodes.
211 mtu:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
212 lower than the outgoing physical interface.
214 [[pvesdn_zone_plugin_evpn]]
218 This is the most complex of all the supported plugins.
220 BGP-EVPN allows you to create a routable layer 3 network. The VNet of EVPN can
221 have an anycast IP address and/or MAC address. The bridge IP is the same on each
222 node, meaning a virtual guest can use this address as gateway.
224 Routing can work across VNets from different zones through a VRF (Virtual
225 Routing and Forwarding) interface.
227 The configuration options specific to EVPN are as follows:
229 VRF VXLAN tag:: This is a VXLAN-ID used for routing interconnect between VNets.
230 It must be different than the VXLAN-ID of the VNets.
232 controller:: An EVPN-controller must to be defined first (see controller plugins
235 VNet MAC address:: A unique, anycast MAC address for all VNets in this zone.
236 Will be auto-generated if not defined.
238 Exit Nodes:: Optional. This is used if you want to define some {pve} nodes as
239 exit gateways from the EVPN network, through the real network. The configured
240 nodes will announce a default route in the EVPN network.
242 Primary Exit Node:: Optional. If you use multiple exit nodes, this forces
243 traffic to a primary exit node, instead of load-balancing on all nodes. This
244 is required if you want to use SNAT or if your upstream router doesn't support
247 Exit Nodes local routing:: Optional. This is a special option if you need to
248 reach a VM/CT service from an exit node. (By default, the exit nodes only
249 allow forwarding traffic between real network and EVPN network).
251 Advertise Subnets:: Optional. If you have silent VMs/CTs (for example, if you
252 have multiple IPs and the anycast gateway doesn't see traffic from theses IPs,
253 the IP addresses won't be able to be reach inside the EVPN network). This
254 option will announce the full subnet in the EVPN network in this case.
256 Disable Arp-Nd Suppression:: Optional. Don't suppress ARP or ND packets.
257 This is required if you use floating IPs in your guest VMs
258 (IP are MAC addresses are being moved between systems).
260 Route-target import:: Optional. Allows you to import a list of external EVPN
261 route targets. Used for cross-DC or different EVPN network interconnects.
263 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
264 less than the maximal MTU of the outgoing physical interface.
267 [[pvesdn_config_vnet]]
271 A `VNet` is, in its basic form, a Linux bridge that will be deployed locally on
272 the node and used for virtual machine communication.
274 The VNet configuration properties are:
276 ID:: An 8 character ID to name and identify a VNet
278 Alias:: Optional longer name, if the ID isn't enough
280 Zone:: The associated zone for this VNet
282 Tag:: The unique VLAN or VXLAN ID
284 VLAN Aware:: Enable adding an extra VLAN tag in the virtual machine or
285 container's vNIC configuration, to allow the guest OS to manage the VLAN's tag.
287 [[pvesdn_config_subnet]]
291 A subnetwork (subnet) allows you to define a specific IP network
292 (IPv4 or IPv6). For each VNet, you can define one or more subnets.
294 A subnet can be used to:
296 * Restrict the IP addresses you can define on a specific VNet
297 * Assign routes/gateways on a VNet in layer 3 zones
298 * Enable SNAT on a VNet in layer 3 zones
299 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
300 * DNS registration through DNS plugins
302 If an IPAM server is associated with the subnet zone, the subnet prefix will be
303 automatically registered in the IPAM.
305 Subnet properties are:
307 ID:: A CIDR network address, for example 10.0.0.0/8
309 Gateway:: The IP address of the network's default gateway. On layer 3 zones
310 (Simple/EVPN plugins), it will be deployed on the VNet.
312 SNAT:: Optional. Enable SNAT for layer 3 zones (Simple/EVPN plugins), for this
313 subnet. The subnet's source IP will be NATted to server's outgoing interface/IP.
314 On EVPN zones, this is only done on EVPN gateway-nodes.
316 Dnszoneprefix:: Optional. Add a prefix to the domain registration, like
317 <hostname>.prefix.<domain>
319 [[pvesdn_config_controllers]]
323 Some zone types need an external controller to manage the VNet control-plane.
324 Currently this is only required for the `bgp-evpn` zone plugin.
326 [[pvesdn_controller_plugin_evpn]]
330 For `BGP-EVPN`, we need a controller to manage the control plane.
331 The currently supported software controller is the "frr" router.
332 You may need to install it on each node where you want to deploy EVPN zones.
335 apt install frr frr-pythontools
338 Configuration options:
340 asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
341 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
342 breaking global routing by mistake.
344 peers:: An IP list of all nodes where you want to communicate for the EVPN
345 (could also be external nodes or route reflectors servers)
348 [[pvesdn_controller_plugin_BGP]]
352 The BGP controller is not used directly by a zone.
353 You can use it to configure FRR to manage BGP peers.
355 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
357 Configuration options:
359 node:: The node of this BGP controller
361 asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
362 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
363 you could break global routing by mistake.
365 peers:: A list of peer IP addresses you want to communicate with using the
366 underlying BGP network.
368 ebgp:: If your peer's remote-AS is different, this enables EBGP.
370 loopback:: Use a loopback or dummy interface as the source of the EVPN network
373 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
374 not directly connected or they use loopback.
376 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
378 [[pvesdn_config_ipam]]
382 IPAM (IP Address Management) tools are used to manage/assign the IP addresses of
383 guests on the network. It can be used to find free IP addresses when you create
384 a VM/CT for example (not yet implemented).
386 An IPAM can be associated with one or more zones, to provide IP addresses
387 for all subnets defined in those zones.
389 [[pvesdn_ipam_plugin_pveipam]]
393 This is the default internal IPAM for your {pve} cluster, if you don't have
394 external IPAM software.
396 [[pvesdn_ipam_plugin_phpipam]]
401 You need to create an application in phpIPAM and add an API token with admin
404 The phpIPAM configuration properties are:
406 url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
408 token:: An API access token
410 section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
411 installations use `sectionid=1` for customers.
413 [[pvesdn_ipam_plugin_netbox]]
417 NetBox is an IP address management (IPAM) and datacenter infrastructure
418 management (DCIM) tool. See the source code repository for details:
419 https://github.com/netbox-community/netbox
421 You need to create an API token in NetBox to use it:
422 https://netbox.readthedocs.io/en/stable/api/authentication
424 The NetBox configuration properties are:
426 url:: The REST API endpoint: `http://yournetbox.domain.com/api`
428 token:: An API access token
430 [[pvesdn_config_dns]]
434 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
435 of your hostname and IP address. A DNS configuration is associated with one or
436 more zones, to provide DNS registration for all the subnet IPs configured for
439 [[pvesdn_dns_plugin_powerdns]]
442 https://doc.powerdns.com/authoritative/http-api/index.html
444 You need to enable the web server and the API in your PowerDNS config:
448 api-key=arandomgeneratedstring
453 The PowerDNS configuration options are:
455 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
457 key:: An API access key
459 ttl:: The default TTL for records
465 [[pvesdn_setup_example_vlan]]
469 TIP: While we show plaintext configuration content here, almost everything
470 should be configurable using the web-interface only.
472 Node1: /etc/network/interfaces
476 iface vmbr0 inet manual
480 bridge-vlan-aware yes
483 #management ip on vlan100
485 iface vmbr0.100 inet static
486 address 192.168.0.1/24
488 source /etc/network/interfaces.d/*
491 Node2: /etc/network/interfaces
495 iface vmbr0 inet manual
499 bridge-vlan-aware yes
502 #management ip on vlan100
504 iface vmbr0.100 inet static
505 address 192.168.0.2/24
507 source /etc/network/interfaces.d/*
510 Create a VLAN zone named `myvlanzone':
517 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
518 `myvlanzone' as its zone.
526 Apply the configuration through the main SDN panel, to create VNets locally on
529 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
531 Use the following network configuration for this VM:
535 iface eth0 inet static
536 address 10.0.3.100/24
539 Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
542 Use the following network configuration for this VM:
546 iface eth0 inet static
547 address 10.0.3.101/24
550 Following this, you should be able to ping between both VMs over that network.
553 [[pvesdn_setup_example_qinq]]
557 TIP: While we show plaintext configuration content here, almost everything
558 should be configurable using the web-interface only.
560 Node1: /etc/network/interfaces
564 iface vmbr0 inet manual
568 bridge-vlan-aware yes
571 #management ip on vlan100
573 iface vmbr0.100 inet static
574 address 192.168.0.1/24
576 source /etc/network/interfaces.d/*
579 Node2: /etc/network/interfaces
583 iface vmbr0 inet manual
587 bridge-vlan-aware yes
590 #management ip on vlan100
592 iface vmbr0.100 inet static
593 address 192.168.0.2/24
595 source /etc/network/interfaces.d/*
598 Create a QinQ zone named `qinqzone1' with service VLAN 20
606 Create another QinQ zone named `qinqzone2' with service VLAN 30
614 Create a VNet named `myvnet1' with customer VLAN-ID 100 on the previously
615 created `qinqzone1' zone.
623 Create a `myvnet2' with customer VLAN-ID 100 on the previously created
632 Apply the configuration on the main SDN web-interface panel to create VNets
633 locally on each nodes.
635 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
637 Use the following network configuration for this VM:
641 iface eth0 inet static
642 address 10.0.3.100/24
645 Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
648 Use the following network configuration for this VM:
652 iface eth0 inet static
653 address 10.0.3.101/24
656 Create a third virtual machine (vm3) on node1, with a vNIC on the other VNet
659 Use the following network configuration for this VM:
663 iface eth0 inet static
664 address 10.0.3.102/24
667 Create another virtual machine (vm4) on node2, with a vNIC on the same VNet
670 Use the following network configuration for this VM:
674 iface eth0 inet static
675 address 10.0.3.103/24
678 Then, you should be able to ping between the VMs 'vm1' and 'vm2', as well as
679 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
680 'vm3' or 'vm4', as they are on a different zone with a different service-vlan.
683 [[pvesdn_setup_example_vxlan]]
687 TIP: While we show plaintext configuration content here, almost everything
688 is configurable through the web-interface.
690 node1: /etc/network/interfaces
694 iface vmbr0 inet static
695 address 192.168.0.1/24
696 gateway 192.168.0.254
702 source /etc/network/interfaces.d/*
705 node2: /etc/network/interfaces
709 iface vmbr0 inet static
710 address 192.168.0.2/24
711 gateway 192.168.0.254
717 source /etc/network/interfaces.d/*
720 node3: /etc/network/interfaces
724 iface vmbr0 inet static
725 address 192.168.0.3/24
726 gateway 192.168.0.254
732 source /etc/network/interfaces.d/*
735 Create a VXLAN zone named `myvxlanzone', using a lower MTU to ensure the extra
736 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
737 the nodes to the peer address list.
741 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
745 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
754 Apply the configuration on the main SDN web-interface panel to create VNets
755 locally on each nodes.
757 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
759 Use the following network configuration for this VM (note the lower MTU).
763 iface eth0 inet static
764 address 10.0.3.100/24
768 Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet
771 Use the following network configuration for this VM:
775 iface eth0 inet static
776 address 10.0.3.101/24
780 Then, you should be able to ping between between 'vm1' and 'vm2'.
783 [[pvesdn_setup_example_evpn]]
787 node1: /etc/network/interfaces
791 iface vmbr0 inet static
792 address 192.168.0.1/24
793 gateway 192.168.0.254
799 source /etc/network/interfaces.d/*
802 node2: /etc/network/interfaces
806 iface vmbr0 inet static
807 address 192.168.0.2/24
808 gateway 192.168.0.254
814 source /etc/network/interfaces.d/*
817 node3: /etc/network/interfaces
821 iface vmbr0 inet static
822 address 192.168.0.3/24
823 gateway 192.168.0.254
829 source /etc/network/interfaces.d/*
832 Create an EVPN controller, using a private ASN number and the above node
838 peers: 192.168.0.1,192.168.0.2,192.168.0.3
841 Create an EVPN zone named `myevpnzone', using the previously created
842 EVPN-controller. Define 'node1' and 'node2' as exit nodes.
847 controller: myevpnctl
849 vnet mac address: 32:F4:05:FE:6C:0A
850 exitnodes: node1,node2
853 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
860 Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on `myvnet1`.
867 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
868 different IPv4 CIDR network.
876 Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
884 Apply the configuration from the main SDN web-interface panel to create VNets
885 locally on each node and generate the FRR config.
887 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
889 Use the following network configuration for this VM:
893 iface eth0 inet static
894 address 10.0.1.100/24
895 gateway 10.0.1.1 #this is the ip of the vnet1
899 Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet
902 Use the following network configuration for this VM:
906 iface eth0 inet static
907 address 10.0.2.100/24
908 gateway 10.0.2.1 #this is the ip of the myvnet2
913 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
915 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
916 will go to the configured 'myvnet2' gateway, then will be routed to the exit
917 nodes ('node1' or 'node2') and from there it will leave those nodes over the
918 default gateway configured on node1 or node2.
920 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
921 networks to node1 and node2 on your external gateway, so that the public network
924 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
925 and 10.0.2.0/24 in this example), will be announced dynamically.
931 VXLAN IPSEC Encryption
932 ~~~~~~~~~~~~~~~~~~~~~~
934 If you need to add encryption on top of a VXLAN, it's possible to do so with
935 IPSEC, through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
936 or 80 bytes (IPv6) to handle encryption.
938 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
939 + 50 (VXLAN) == 1500).
943 apt install strongswan
946 Add configuration to `/etc/ipsec.conf'. We only need to encrypt traffic from
947 the VXLAN UDP port '4789'.
951 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
953 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
956 rightsubnet=%dynamic[udp/4789]
963 leftsubnet=%dynamic[udp/4789]
969 Then generate a pre-shared key with:
972 openssl rand -base64 128
975 and add the key to `/etc/ipsec.secrets', so that the file contents looks like:
978 : PSK <generatedbase64key>
981 You need to copy the PSK and the configuration onto the other nodes.