2 Software Defined Network
3 ========================
8 The **S**oftware **D**efined **N**etwork (SDN) feature allows you to create
9 virtual networks (VNets) at the datacenter level.
11 WARNING: SDN is currently an **experimental feature** in {pve}. This
12 documentation for it is also still under development. Ask on our
13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
16 [[pvesdn_installation]]
20 To enable the experimental Software Defined Network (SDN) integration, you need
21 to install the `libpve-network-perl` and `ifupdown2` packages on every node:
25 apt install libpve-network-perl ifupdown2
28 NOTE: {pve} version 7 and above come installed with ifupdown2.
30 After this, you need to add the following line to the end of the
31 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
32 included and activated.
35 source /etc/network/interfaces.d/*
42 The {pve} SDN allows for separation and fine-grained control of virtual guest
43 networks, using flexible, software-controlled configurations.
45 Separation is managed through zones, where a zone is its own virtual separated
46 network area. A 'VNet' is a type of a virtual network connected to a zone.
47 Depending on which type or plugin the zone uses, it can behave differently and
48 offer different features, advantages, and disadvantages. Normally, a 'VNet'
49 appears as a common Linux bridge with either a VLAN or 'VXLAN' tag, however,
50 some can also use layer 3 routing for control. 'VNets' are deployed locally on
51 each node, after being configured from the cluster-wide datacenter SDN
52 administration interface.
58 Configuration is done at the datacenter (cluster-wide) level and is saved in
59 files located in the shared configuration file system:
62 On the web-interface, SDN features 3 main sections:
64 * SDN: An overview of the SDN state
66 * Zones: Create and manage the virtually separated network zones
68 * VNets: Create virtual network bridges and manage subnets
70 In addition to this, the following options are offered:
72 * Controller: For controlling layer 3 routing in complex setups
74 * Subnets: Used to defined IP networks on VNets
76 * IPAM: Enables the use of external tools for IP address management (guest
79 * DNS: Define a DNS server API for registering virtual guests' hostname and IP
82 [[pvesdn_config_main_sdn]]
87 This is the main status panel. Here you can see the deployment status of zones
90 The 'Apply' button is used to push and reload local configuration on all cluster
94 [[pvesdn_local_deployment_monitoring]]
95 Local Deployment Monitoring
96 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
98 After applying the configuration through the main SDN panel,
99 the local network configuration is generated locally on each node in
100 the file `/etc/network/interfaces.d/sdn`, and reloaded with ifupdown2.
102 You can monitor the status of local zones and VNets through the main tree.
105 [[pvesdn_config_zone]]
109 A zone defines a virtually separated network. Zones can be restricted to
110 specific nodes and assigned permissions, in order to restrict users to a certain
111 zone and its contained VNets.
113 Different technologies can be used for separation:
115 * VLAN: Virtual LANs are the classic method of subdividing a LAN
117 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
119 * VXLAN: Layer2 VXLAN
121 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
123 * EVPN (BGP EVPN): VXLAN using layer 3 border gateway protocol (BGP) routing
128 The following options are available for all zone types:
130 nodes:: The nodes which the zone and associated VNets should be deployed on
132 ipam:: Optional. Use an IP Address Management (IPAM) tool to manage IPs in the
135 dns:: Optional. DNS API server.
137 reversedns:: Optional. Reverse DNS API server.
139 dnszone:: Optional. DNS domain name. Used to register hostnames, such as
140 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server.
143 [[pvesdn_zone_plugin_simple]]
147 This is the simplest plugin. It will create an isolated VNet bridge.
148 This bridge is not linked to a physical interface, and VM traffic is only
149 local between the node(s).
150 It can also be used in NAT or routed setups.
152 [[pvesdn_zone_plugin_vlan]]
156 This plugin reuses an existing local Linux or OVS bridge, and manages the VLANs
157 on it. The benefit of using the SDN module is that you can create different
158 zones with specific VNet VLAN tags, and restrict virtual machines to separated
161 Specific `VLAN` configuration options:
163 bridge:: Reuse this local bridge or OVS switch, already configured on *each*
166 [[pvesdn_zone_plugin_qinq]]
170 QinQ also known as VLAN stacking, wherein the first VLAN tag is defined for the
171 zone (the 'service-vlan'), and the second VLAN tag is defined for the
174 NOTE: Your physical network switches must support stacked VLANs for this
177 Below are the configuration options specific to QinQ:
179 bridge:: A local, VLAN-aware bridge that is already configured on each local
182 service vlan:: The main VLAN tag of this zone
184 service vlan protocol:: Allows you to choose between an 802.1q (default) or
185 802.1ad service VLAN type.
187 mtu:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
188 For example, you must reduce the MTU to `1496` if you physical interface MTU is
191 [[pvesdn_zone_plugin_vxlan]]
195 The VXLAN plugin establishes a tunnel (overlay) on top of an existing
196 network (underlay). This encapsulates layer 2 Ethernet frames within layer
197 4 UDP datagrams, using `4789` as the default destination port. You can, for
198 example, create a private IPv4 VXLAN network on top of public internet network
201 This is a layer 2 tunnel only, so no routing between different VNets is
204 Each VNet will have a specific VXLAN ID in the range 1 - 16777215.
206 Specific EVPN configuration options:
208 peers address list:: A list of IP addresses from each node through which you
209 want to communicate. Can also be external nodes.
211 mtu:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
212 lower than the outgoing physical interface.
214 [[pvesdn_zone_plugin_evpn]]
218 This is the most complex of all the supported plugins.
220 BGP-EVPN allows you to create a routable layer 3 network. The VNet of EVPN can
221 have an anycast IP address and/or MAC address. The bridge IP is the same on each
222 node, meaning a virtual guest can use this address as gateway.
224 Routing can work across VNets from different zones through a VRF (Virtual
225 Routing and Forwarding) interface.
227 The configuration options specific to EVPN are as follows:
229 VRF VXLAN tag:: This is a VXLAN-ID used for routing interconnect between VNets.
230 It must be different than the VXLAN-ID of the VNets.
232 controller:: An EVPN-controller must to be defined first (see controller plugins
235 VNet MAC address:: A unique, anycast MAC address for all VNets in this zone.
236 Will be auto-generated if not defined.
238 Exit Nodes:: Optional. This is used if you want to define some {pve} nodes as
239 exit gateways from the EVPN network, through the real network. The configured
240 nodes will announce a default route in the EVPN network.
242 Primary Exit Node:: Optional. If you use multiple exit nodes, this forces
243 traffic to a primary exit node, instead of load-balancing on all nodes. This
244 is required if you want to use SNAT or if your upstream router doesn't support
247 Exit Nodes local routing:: Optional. This is a special option if you need to
248 reach a VM/CT service from an exit node. (By default, the exit nodes only
249 allow forwarding traffic between real network and EVPN network).
251 Advertise Subnets:: Optional. If you have silent VMs/CTs (for example, if you
252 have multiple IPs and the anycast gateway doesn't see traffic from theses IPs,
253 the IP addresses won't be able to be reach inside the EVPN network). This
254 option will announce the full subnet in the EVPN network in this case.
256 Disable Arp-Nd Suppression:: Optional. Don't suppress ARP or ND packets.
257 This is required if you use floating IPs in your guest VMs
258 (IP are MAC addresses are being moved between systems).
260 Route-target import:: Optional. Allows you to import a list of external EVPN
261 route targets. Used for cross-DC or different EVPN network interconnects.
263 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
264 less than the maximal MTU of the outgoing physical interface.
267 [[pvesdn_config_vnet]]
271 A `VNet` is, in its basic form, a Linux bridge that will be deployed locally on
272 the node and used for virtual machine communication.
274 The VNet configuration properties are:
276 ID:: An 8 character ID to name and identify a VNet
278 Alias:: Optional longer name, if the ID isn't enough
280 Zone:: The associated zone for this VNet
282 Tag:: The unique VLAN or VXLAN ID
284 VLAN Aware:: Enable adding an extra VLAN tag in the virtual machine or
285 container's vNIC configuration, to allow the guest OS to manage the VLAN's tag.
287 [[pvesdn_config_subnet]]
291 A subnetwork (subnet) allows you to define a specific IP network
292 (IPv4 or IPv6). For each VNet, you can define one or more subnets.
294 A subnet can be used to:
296 * Restrict the IP addresses you can define on a specific VNet
297 * Assign routes/gateways on a VNet in layer 3 zones
298 * Enable SNAT on a VNet in layer 3 zones
299 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
300 * DNS registration through DNS plugins
302 If an IPAM server is associated with the subnet zone, the subnet prefix will be
303 automatically registered in the IPAM.
305 Subnet properties are:
307 ID:: A CIDR network address, for example 10.0.0.0/8
309 Gateway:: The IP address of the network's default gateway. On layer 3 zones
310 (Simple/EVPN plugins), it will be deployed on the VNet.
312 SNAT:: Optional. Enable SNAT for layer 3 zones (Simple/EVPN plugins), for this
313 subnet. The subnet's source IP will be NATted to server's outgoing interface/IP.
314 On EVPN zones, this is only done on EVPN gateway-nodes.
316 Dnszoneprefix:: Optional. Add a prefix to the domain registration, like
317 <hostname>.prefix.<domain>
319 [[pvesdn_config_controllers]]
323 Some zone types need an external controller to manage the VNet control-plane.
324 Currently this is only required for the `bgp-evpn` zone plugin.
326 [[pvesdn_controller_plugin_evpn]]
330 For `BGP-EVPN`, we need a controller to manage the control plane.
331 The currently supported software controller is the "frr" router.
332 You may need to install it on each node where you want to deploy EVPN zones.
335 apt install frr frr-pythontools
338 Configuration options:
340 asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
341 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
342 breaking global routing by mistake.
344 peers:: An IP list of all nodes where you want to communicate for the EVPN
345 (could also be external nodes or route reflectors servers)
348 [[pvesdn_controller_plugin_BGP]]
352 The BGP controller is not used directly by a zone.
353 You can use it to configure FRR to manage BGP peers.
355 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
356 It can also be used to export EVPN routes to an external BGP peer.
358 NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
361 Configuration options:
363 node:: The node of this BGP controller
365 asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
366 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
367 you could break global routing by mistake.
369 peers:: A list of peer IP addresses you want to communicate with using the
370 underlying BGP network.
372 ebgp:: If your peer's remote-AS is different, this enables EBGP.
374 loopback:: Use a loopback or dummy interface as the source of the EVPN network
377 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
378 not directly connected or they use loopback.
380 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
383 [[pvesdn_controller_plugin_ISIS]]
387 The ISIS controller is not used directly by a zone.
388 You can use it to configure FRR to export evpn routes to an ISIS domain.
390 Configuration options:
392 node:: The node of this ISIS controller.
394 domain:: A unique ISIS domain.
396 network entity title:: A Unique ISIS network address that identifies this node.
398 interfaces:: A list of physical interface(s) used by ISIS.
400 loopback:: Use a loopback or dummy interface as the source of the EVPN network
403 [[pvesdn_config_ipam]]
407 IPAM (IP Address Management) tools are used to manage/assign the IP addresses of
408 guests on the network. It can be used to find free IP addresses when you create
409 a VM/CT for example (not yet implemented).
411 An IPAM can be associated with one or more zones, to provide IP addresses
412 for all subnets defined in those zones.
414 [[pvesdn_ipam_plugin_pveipam]]
418 This is the default internal IPAM for your {pve} cluster, if you don't have
419 external IPAM software.
421 [[pvesdn_ipam_plugin_phpipam]]
426 You need to create an application in phpIPAM and add an API token with admin
429 The phpIPAM configuration properties are:
431 url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
433 token:: An API access token
435 section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
436 installations use `sectionid=1` for customers.
438 [[pvesdn_ipam_plugin_netbox]]
442 NetBox is an IP address management (IPAM) and datacenter infrastructure
443 management (DCIM) tool. See the source code repository for details:
444 https://github.com/netbox-community/netbox
446 You need to create an API token in NetBox to use it:
447 https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
449 The NetBox configuration properties are:
451 url:: The REST API endpoint: `http://yournetbox.domain.com/api`
453 token:: An API access token
455 [[pvesdn_config_dns]]
459 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
460 of your hostname and IP address. A DNS configuration is associated with one or
461 more zones, to provide DNS registration for all the subnet IPs configured for
464 [[pvesdn_dns_plugin_powerdns]]
467 https://doc.powerdns.com/authoritative/http-api/index.html
469 You need to enable the web server and the API in your PowerDNS config:
473 api-key=arandomgeneratedstring
478 The PowerDNS configuration options are:
480 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
482 key:: An API access key
484 ttl:: The default TTL for records
490 [[pvesdn_setup_example_vlan]]
494 TIP: While we show plaintext configuration content here, almost everything
495 should be configurable using the web-interface only.
497 Node1: /etc/network/interfaces
501 iface vmbr0 inet manual
505 bridge-vlan-aware yes
508 #management ip on vlan100
510 iface vmbr0.100 inet static
511 address 192.168.0.1/24
513 source /etc/network/interfaces.d/*
516 Node2: /etc/network/interfaces
520 iface vmbr0 inet manual
524 bridge-vlan-aware yes
527 #management ip on vlan100
529 iface vmbr0.100 inet static
530 address 192.168.0.2/24
532 source /etc/network/interfaces.d/*
535 Create a VLAN zone named `myvlanzone':
542 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
543 `myvlanzone' as its zone.
551 Apply the configuration through the main SDN panel, to create VNets locally on
554 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
556 Use the following network configuration for this VM:
560 iface eth0 inet static
561 address 10.0.3.100/24
564 Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
567 Use the following network configuration for this VM:
571 iface eth0 inet static
572 address 10.0.3.101/24
575 Following this, you should be able to ping between both VMs over that network.
578 [[pvesdn_setup_example_qinq]]
582 TIP: While we show plaintext configuration content here, almost everything
583 should be configurable using the web-interface only.
585 Node1: /etc/network/interfaces
589 iface vmbr0 inet manual
593 bridge-vlan-aware yes
596 #management ip on vlan100
598 iface vmbr0.100 inet static
599 address 192.168.0.1/24
601 source /etc/network/interfaces.d/*
604 Node2: /etc/network/interfaces
608 iface vmbr0 inet manual
612 bridge-vlan-aware yes
615 #management ip on vlan100
617 iface vmbr0.100 inet static
618 address 192.168.0.2/24
620 source /etc/network/interfaces.d/*
623 Create a QinQ zone named `qinqzone1' with service VLAN 20
631 Create another QinQ zone named `qinqzone2' with service VLAN 30
639 Create a VNet named `myvnet1' with customer VLAN-ID 100 on the previously
640 created `qinqzone1' zone.
648 Create a `myvnet2' with customer VLAN-ID 100 on the previously created
657 Apply the configuration on the main SDN web-interface panel to create VNets
658 locally on each nodes.
660 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
662 Use the following network configuration for this VM:
666 iface eth0 inet static
667 address 10.0.3.100/24
670 Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
673 Use the following network configuration for this VM:
677 iface eth0 inet static
678 address 10.0.3.101/24
681 Create a third virtual machine (vm3) on node1, with a vNIC on the other VNet
684 Use the following network configuration for this VM:
688 iface eth0 inet static
689 address 10.0.3.102/24
692 Create another virtual machine (vm4) on node2, with a vNIC on the same VNet
695 Use the following network configuration for this VM:
699 iface eth0 inet static
700 address 10.0.3.103/24
703 Then, you should be able to ping between the VMs 'vm1' and 'vm2', as well as
704 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
705 'vm3' or 'vm4', as they are on a different zone with a different service-vlan.
708 [[pvesdn_setup_example_vxlan]]
712 TIP: While we show plaintext configuration content here, almost everything
713 is configurable through the web-interface.
715 node1: /etc/network/interfaces
719 iface vmbr0 inet static
720 address 192.168.0.1/24
721 gateway 192.168.0.254
727 source /etc/network/interfaces.d/*
730 node2: /etc/network/interfaces
734 iface vmbr0 inet static
735 address 192.168.0.2/24
736 gateway 192.168.0.254
742 source /etc/network/interfaces.d/*
745 node3: /etc/network/interfaces
749 iface vmbr0 inet static
750 address 192.168.0.3/24
751 gateway 192.168.0.254
757 source /etc/network/interfaces.d/*
760 Create a VXLAN zone named `myvxlanzone', using a lower MTU to ensure the extra
761 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
762 the nodes to the peer address list.
766 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
770 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
779 Apply the configuration on the main SDN web-interface panel to create VNets
780 locally on each nodes.
782 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
784 Use the following network configuration for this VM (note the lower MTU).
788 iface eth0 inet static
789 address 10.0.3.100/24
793 Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet
796 Use the following network configuration for this VM:
800 iface eth0 inet static
801 address 10.0.3.101/24
805 Then, you should be able to ping between between 'vm1' and 'vm2'.
808 [[pvesdn_setup_example_evpn]]
812 node1: /etc/network/interfaces
816 iface vmbr0 inet static
817 address 192.168.0.1/24
818 gateway 192.168.0.254
824 source /etc/network/interfaces.d/*
827 node2: /etc/network/interfaces
831 iface vmbr0 inet static
832 address 192.168.0.2/24
833 gateway 192.168.0.254
839 source /etc/network/interfaces.d/*
842 node3: /etc/network/interfaces
846 iface vmbr0 inet static
847 address 192.168.0.3/24
848 gateway 192.168.0.254
854 source /etc/network/interfaces.d/*
857 Create an EVPN controller, using a private ASN number and the above node
863 peers: 192.168.0.1,192.168.0.2,192.168.0.3
866 Create an EVPN zone named `myevpnzone', using the previously created
867 EVPN-controller. Define 'node1' and 'node2' as exit nodes.
872 controller: myevpnctl
874 vnet mac address: 32:F4:05:FE:6C:0A
875 exitnodes: node1,node2
878 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
885 Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on `myvnet1`.
892 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
893 different IPv4 CIDR network.
901 Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
909 Apply the configuration from the main SDN web-interface panel to create VNets
910 locally on each node and generate the FRR config.
912 Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
914 Use the following network configuration for this VM:
918 iface eth0 inet static
919 address 10.0.1.100/24
920 gateway 10.0.1.1 #this is the ip of the vnet1
924 Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet
927 Use the following network configuration for this VM:
931 iface eth0 inet static
932 address 10.0.2.100/24
933 gateway 10.0.2.1 #this is the ip of the myvnet2
938 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
940 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
941 will go to the configured 'myvnet2' gateway, then will be routed to the exit
942 nodes ('node1' or 'node2') and from there it will leave those nodes over the
943 default gateway configured on node1 or node2.
945 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
946 networks to node1 and node2 on your external gateway, so that the public network
949 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
950 and 10.0.2.0/24 in this example), will be announced dynamically.
956 Multiple EVPN Exit Nodes
957 ~~~~~~~~~~~~~~~~~~~~~~~~
959 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
960 Reverse Path Filter) option, because packets can arrive at one node but go out
963 .sysctl.conf disabling `rp_filter`
965 net.ipv4.conf.default.rp_filter=0
966 net.ipv4.conf.all.rp_filter=0
969 VXLAN IPSEC Encryption
970 ~~~~~~~~~~~~~~~~~~~~~~
972 If you need to add encryption on top of a VXLAN, it's possible to do so with
973 IPSEC, through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
974 or 80 bytes (IPv6) to handle encryption.
976 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
977 + 50 (VXLAN) == 1500).
981 apt install strongswan
984 Add configuration to `/etc/ipsec.conf'. We only need to encrypt traffic from
985 the VXLAN UDP port '4789'.
989 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
991 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
994 rightsubnet=%dynamic[udp/4789]
1001 leftsubnet=%dynamic[udp/4789]
1007 Then generate a pre-shared key with:
1010 openssl rand -base64 128
1013 and add the key to `/etc/ipsec.secrets', so that the file contents looks like:
1016 : PSK <generatedbase64key>
1019 You need to copy the PSK and the configuration onto the other nodes.