2 Software Defined Network
3 ========================
8 The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
9 virtual networks (vnets) at datacenter level.
11 WARNING: SDN is currently an **experimental feature** in {pve}. This
12 Documentation for it is also still under development, ask on our
13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
16 [[pvesdn_installation]]
20 To enable the experimental SDN integration, you need to install the
21 `libpve-network-perl` and `ifupdown2` package on every node:
25 apt install libpve-network-perl ifupdown2
28 After that you need to add the following line:
31 source /etc/network/interfaces.d/*
33 at the end of the `/etc/network/interfaces` configuration file, so that the SDN
34 config gets included and activated.
40 The {pve} SDN allows separation and fine grained control of Virtual Guests
41 networks, using flexible software controlled configurations.
43 Separation consists of zones, a zone is it's own virtual separated network area.
44 A 'VNet' is a type of a virtual network connected to a zone. Depending on which
45 type or plugin the zone uses it can behave differently and offer different
46 features, advantages or disadvantages.
47 Normally a 'VNet' shows up as a common Linux bridge with either a VLAN or
48 'VXLAN' tag, but some can also use layer 3 routing for control.
49 The 'VNets' are deployed locally on each node, after configuration was committed
50 from the cluster-wide datacenter SDN administration interface.
56 The configuration is done at datacenter (cluster-wide) level, it will be saved
57 in configuration files located in the shared configuration file system:
60 On the web-interface SDN feature have 3 main sections for the configuration
62 * SDN: a overview of the SDN state
64 * Zones: Create and manage the virtual separated network Zones
66 * VNets: Create virtual network bridges + subnets management.
70 * Controller: For complex setups to control Layer 3 routing
72 * Sub-nets: Used to defined ip networks on VNets.
74 * IPAM: Allow to use external tools for IP address management (guest IPs)
76 * DNS: Allow to define a DNS server api for registering a virtual guests
77 hostname and IP-addresses
79 [[pvesdn_config_main_sdn]]
84 This is the main status panel. Here you can see deployment status of zones on
87 There is an 'Apply' button, to push and reload local configuration on all
91 [[pvesdn_local_deployment_monitoring]]
92 Local Deployment Monitoring
93 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
95 After applying the configuration through the main SDN web-interface panel,
96 the local network configuration is generated locally on each node in
97 `/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
99 You can monitor the status of local zones and vnets through the main tree.
102 [[pvesdn_config_zone]]
106 A zone will define a virtually separated network.
108 It can use different technologies for separation:
110 * VLAN: Virtual LANs are the classic method to sub-divide a LAN
112 * QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
114 * VXLAN: (layer2 vxlan)
116 * Simple: Isolated Bridge, simple l3 routing bridge (NAT)
118 * bgp-evpn: vxlan using layer3 border gateway protocol routing
120 You can restrict a zone to specific nodes.
122 It's also possible to add permissions on a zone, to restrict user to use only a
123 specific zone and only the VNets in that zone
128 The following options are available for all zone types.
130 nodes:: Deploy and allow to use a VNets configured for this Zone only on these
133 ipam:: Optional, if you want to use an ipam tool to manage ips in this zone
135 dns:: Optional, dns api server.
137 reversedns:: Optional, reverse dns api server.
139 dnszone:: Optional, dns domain name. Use to register hostname like
140 `<hostname>.<domain>`. The dns zone need to be already existing in dns server.
143 [[pvesdn_zone_plugin_simple]]
147 This is the simplest plugin, it will create an isolated vnet bridge.
148 This bridge is not linked to physical interfaces, VM traffic is only
149 local to the node(s).
150 It can be also used for NAT or routed setup.
152 [[pvesdn_zone_plugin_vlan]]
156 This plugin will reuse an existing local Linux or OVS bridge,
157 and manage VLANs on it.
158 The benefit of using SDN module, is that you can create different zones with
159 specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
161 Specific `VLAN` configuration options:
163 bridge:: Reuse this local bridge or OVS switch, already
164 configured on *each* local node.
166 [[pvesdn_zone_plugin_qinq]]
170 QinQ is stacked VLAN. The first VLAN tag defined for the zone
171 (so called 'service-vlan'), and the second VLAN tag defined for the vnets
173 NOTE: Your physical network switches must support stacked VLANs!
175 Specific QinQ configuration options:
177 bridge:: A local VLAN-aware bridge already configured on each local node
179 service vlan:: The main VLAN tag of this zone
181 service vlan protocol:: allow to define a 802.1q (default) or 802.1ad service vlan type.
183 mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
184 For example, you reduce the MTU to `1496` if you physical interface MTU is
187 [[pvesdn_zone_plugin_vxlan]]
191 The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
192 network (named underlay). It encapsulate layer 2 Ethernet frames within layer
193 4 UDP datagrams, using `4789` as the default destination port. You can, for
194 example, create a private IPv4 VXLAN network on top of public internet network
196 This is a layer2 tunnel only, no routing between different VNets is possible.
198 Each VNet will have use specific VXLAN id from the range (1 - 16777215).
200 Specific EVPN configuration options:
202 peers address list:: A list of IPs from all nodes through which you want to
203 communicate. Can also be external nodes.
205 mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
206 lower than the outgoing physical interface.
208 [[pvesdn_zone_plugin_evpn]]
212 This is the most complex of all supported plugins.
214 BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
215 have an anycast IP-address and or MAC-address. The bridge IP is the same on each
216 node, with this a virtual guest can use that address as gateway.
218 Routing can work across VNets from different zones through a VRF (Virtual
219 Routing and Forwarding) interface.
221 Specific EVPN configuration options:
223 VRF VXLAN tag:: This is a vxlan-id used for routing interconnect between vnets,
224 it must be different than VXLAN-id of VNets
226 controller:: an EVPN-controller need to be defined first (see controller
229 VNet MAC address:: A unique anycast MAC address for all VNets in this zone.
230 Will be auto-generated if not defined.
232 Exit Nodes:: This is used if you want to define some proxmox nodes, as exit
233 gateway from evpn network through real network. The configured nodes will
234 announce a default route in the EVPN network.
236 Advertise Subnets:: Optional. If you have silent vms/CT (for example, multiples
237 ips by interfaces, and the anycast gateway don't see traffic from theses ips,
238 the ips addresses won't be able to be reach inside the evpn network). This
239 option will announce the full subnet in the evpn network in this case.
241 Exit Nodes local routing:: Optional. This is a special option if you need to
242 reach a vm/ct service from an exit node. (By default, the exit nodes only
243 allow forwarding traffic between real network and evpn network).
245 MTU:: because VXLAN encapsulation use 50 bytes, the MTU needs to be 50 bytes
246 lower than the maximal MTU of the outgoing physical interface.
249 [[pvesdn_config_vnet]]
253 A `VNet` is in its basic form just a Linux bridge that will be deployed locally
254 on the node and used for Virtual Machine communication.
258 ID:: a 8 characters ID to name and identify a VNet
260 Alias:: Optional longer name, if the ID isn't enough
262 Zone:: The associated zone for this VNet
264 Tag:: The unique VLAN or VXLAN id
266 VLAN Aware:: Allow to add an extra VLAN tag in the virtual machine or
267 container vNIC configurations or allow the guest OS to manage the VLAN's tag.
269 [[pvesdn_config_subnet]]
274 A sub-network (subnet or sub-net) allows you to define a specific IP network
275 (IPv4 or IPv6). For each VNET, you can define one or more subnets.
277 A subnet can be used to:
279 * restrict IP-addresses you can define on a specific VNET
280 * assign routes/gateway on a VNET in layer 3 zones
281 * enable SNAT on a VNET in layer 3 zones
282 * auto assign IPs on virtual guests (VM or CT) through IPAM plugin
283 * DNS registration through DNS plugins
285 If an IPAM server is associated to the subnet zone, the subnet prefix will be
286 automatically registered in the IPAM.
289 Subnet properties are:
291 ID:: a cidr network address. Ex: 10.0.0.0/8
293 Gateway:: ip address for the default gateway of the network.
294 On layer3 zones (simple/evpn plugins), it'll be deployed on the vnet.
296 Snat:: Optional, Enable Snat for layer3 zones (simple/evpn plugins) for this subnet.
297 The subnet source ip will be natted to server outgoing interface/ip.
298 On evpn zone, it's done only on evpn gateway-nodes.
300 Dnszoneprefix:: Optional, add a prefix to domain registration, like <hostname>.prefix.<domain>
303 [[pvesdn_config_controllers]]
307 Some zone types need an external controller to manage the VNet control-plane.
308 Currently this is only required for the `bgp-evpn` zone plugin.
310 [[pvesdn_controller_plugin_evpn]]
314 For `BGP-EVPN`, we need a controller to manage the control plane.
315 The currently supported software controller is the "frr" router.
316 You may need to install it on each node where you want to deploy EVPN zones.
319 apt install frr frr-pythontools
322 Configuration options:
324 asn:: A unique BGP ASN number. It's highly recommended to use private ASN
325 number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
326 breaking, or get broken, by global routing by mistake.
328 peers:: An ip list of all nodes where you want to communicate for the EVPN (could be also
329 external nodes or route reflectors servers)
332 [[pvesdn_controller_plugin_BGP]]
336 The bgp controller is not used directly by a zone.
337 You can used it to configure frr to manage bgp peers.
339 For BGP-evpn, it can be use to define a different ASN by node, so doing EBGP.
341 Configuration options:
343 node:: The node of this BGP controller
345 asn:: A unique BGP ASN number. It's highly recommended to use private ASN
346 number from the range (64512 - 65534) or (4200000000 - 4294967294), as else
347 you could end up breaking, or get broken, by global routing by mistake.
349 peers:: An IP list of peers you want to communicate with for the underlying
352 ebgp:: If your peer's remote-AS is different, it's enabling EBGP.
354 loopback:: If you want to use a loopback or dummy interface as source for the
355 evpn network. (for multipath)
357 ebgp-mutltihop:: if the peers are not directly connected or use loopback, you can increase the
358 number of hops to reach them.
360 [[pvesdn_config_ipam]]
363 IPAM (IP address management) tools, are used to manage/assign ips on your devices on the network.
364 It can be used to find free ip address when you create a vm/ct for example (not yet implemented).
366 An IPAM is associated to 1 or multiple zones, to provide ip addresses for all subnets defined in this zone.
369 [[pvesdn_ipam_plugin_pveipam]]
373 This is the default internal IPAM for your proxmox cluster if you don't have
374 external ipam software
376 [[pvesdn_ipam_plugin_phpipam]]
381 You need to create an application in phpipam, and add an api token with admin
384 phpIPAM properties are:
386 url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
387 token:: An API access token
388 section:: An integer ID. Sections are group of subnets in phpIPAM. Default
389 installations use `sectionid=1` for customers.
391 [[pvesdn_ipam_plugin_netbox]]
395 NetBox is an IP address management (IPAM) and data center infrastructure
396 management (DCIM) tool, see the source code repository for details:
397 https://github.com/netbox-community/netbox
399 You need to create an api token in netbox
400 https://netbox.readthedocs.io/en/stable/api/authentication
402 NetBox properties are:
404 url:: The REST API endpoint: `http://yournetbox.domain.com/api`
405 token:: An API access token
407 [[pvesdn_config_dns]]
411 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
412 of your hostname and IP-address. A DNS configuration is associated with one or
413 more zones, to provide DNS registration for all the sub-net IPs configured for
416 [[pvesdn_dns_plugin_powerdns]]
419 https://doc.powerdns.com/authoritative/http-api/index.html
421 You need to enable the webserver and the API in your PowerDNS config:
425 api-key=arandomgeneratedstring
430 Powerdns properties are:
432 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
433 key:: An API access key
434 ttl:: The default TTL for records
440 [[pvesdn_setup_example_vlan]]
444 TIP: While we show plain configuration content here, almost everything should
445 be configurable using the web-interface only.
447 Node1: /etc/network/interfaces
451 iface vmbr0 inet manual
455 bridge-vlan-aware yes
458 #management ip on vlan100
460 iface vmbr0.100 inet static
461 address 192.168.0.1/24
463 source /etc/network/interfaces.d/*
466 Node2: /etc/network/interfaces
470 iface vmbr0 inet manual
474 bridge-vlan-aware yes
477 #management ip on vlan100
479 iface vmbr0.100 inet static
480 address 192.168.0.2/24
482 source /etc/network/interfaces.d/*
485 Create a VLAN zone named `myvlanzone':
492 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
493 `myvlanzone' as it's zone.
501 Apply the configuration through the main SDN panel, to create VNets locally on
504 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
506 Use the following network configuration for this VM:
510 iface eth0 inet static
511 address 10.0.3.100/24
514 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
517 Use the following network configuration for this VM:
521 iface eth0 inet static
522 address 10.0.3.101/24
525 Then, you should be able to ping between both VMs over that network.
528 [[pvesdn_setup_example_qinq]]
532 TIP: While we show plain configuration content here, almost everything should
533 be configurable using the web-interface only.
535 Node1: /etc/network/interfaces
539 iface vmbr0 inet manual
543 bridge-vlan-aware yes
546 #management ip on vlan100
548 iface vmbr0.100 inet static
549 address 192.168.0.1/24
551 source /etc/network/interfaces.d/*
554 Node2: /etc/network/interfaces
558 iface vmbr0 inet manual
562 bridge-vlan-aware yes
565 #management ip on vlan100
567 iface vmbr0.100 inet static
568 address 192.168.0.2/24
570 source /etc/network/interfaces.d/*
573 Create an QinQ zone named `qinqzone1' with service VLAN 20
581 Create another QinQ zone named `qinqzone2' with service VLAN 30
589 Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
590 created `qinqzone1' zone.
598 Create a `myvnet2' with customer VLAN-id 100 on the previously created
607 Apply the configuration on the main SDN web-interface panel to create VNets
608 locally on each nodes.
610 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
612 Use the following network configuration for this VM:
616 iface eth0 inet static
617 address 10.0.3.100/24
620 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
623 Use the following network configuration for this VM:
627 iface eth0 inet static
628 address 10.0.3.101/24
631 Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
634 Use the following network configuration for this VM:
638 iface eth0 inet static
639 address 10.0.3.102/24
642 Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
645 Use the following network configuration for this VM:
649 iface eth0 inet static
650 address 10.0.3.103/24
653 Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
654 between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
655 or 'vm4', as they are on a different zone with different service-vlan.
658 [[pvesdn_setup_example_vxlan]]
662 TIP: While we show plain configuration content here, almost everything should
663 be configurable using the web-interface only.
665 node1: /etc/network/interfaces
669 iface vmbr0 inet static
670 address 192.168.0.1/24
671 gateway 192.168.0.254
677 source /etc/network/interfaces.d/*
680 node2: /etc/network/interfaces
684 iface vmbr0 inet static
685 address 192.168.0.2/24
686 gateway 192.168.0.254
692 source /etc/network/interfaces.d/*
695 node3: /etc/network/interfaces
699 iface vmbr0 inet static
700 address 192.168.0.3/24
701 gateway 192.168.0.254
707 source /etc/network/interfaces.d/*
710 Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
711 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
712 the nodes as peer address list.
716 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
720 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
729 Apply the configuration on the main SDN web-interface panel to create VNets
730 locally on each nodes.
732 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
734 Use the following network configuration for this VM, note the lower MTU here.
738 iface eth0 inet static
739 address 10.0.3.100/24
743 Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
746 Use the following network configuration for this VM:
750 iface eth0 inet static
751 address 10.0.3.101/24
755 Then, you should be able to ping between between 'vm1' and 'vm2'.
758 [[pvesdn_setup_example_evpn]]
762 node1: /etc/network/interfaces
766 iface vmbr0 inet static
767 address 192.168.0.1/24
768 gateway 192.168.0.254
774 source /etc/network/interfaces.d/*
777 node2: /etc/network/interfaces
781 iface vmbr0 inet static
782 address 192.168.0.2/24
783 gateway 192.168.0.254
789 source /etc/network/interfaces.d/*
792 node3: /etc/network/interfaces
796 iface vmbr0 inet static
797 address 192.168.0.3/24
798 gateway 192.168.0.254
804 source /etc/network/interfaces.d/*
807 Create a EVPN controller, using a private ASN number and above node addreesses
813 peers: 192.168.0.1,192.168.0.2,192.168.0.3
816 Create an EVPN zone named `myevpnzone' using the previously created
817 EVPN-controller Define 'node1' and 'node2' as exit nodes.
822 controller: myevpnctl
824 vnet mac address: 32:F4:05:FE:6C:0A
825 exitnodes: node1,node2
828 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
835 Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on vnet1
842 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
843 different IPv4 CIDR network.
851 Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
859 Apply the configuration on the main SDN web-interface panel to create VNets
860 locally on each nodes and generate the FRR config.
862 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
864 Use the following network configuration for this VM:
868 iface eth0 inet static
869 address 10.0.1.100/24
870 gateway 10.0.1.1 #this is the ip of the vnet1
874 Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
877 Use the following network configuration for this VM:
881 iface eth0 inet static
882 address 10.0.2.100/24
883 gateway 10.0.2.1 #this is the ip of the vnet2
888 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
890 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
891 will go to the configured 'myvnet2' gateway, then will be routed to the exit
892 nodes ('node1' or 'node2') and from there it will leave those nodes over the
893 default gateway configured on node1 or node2.
895 NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
896 '10.0.2.0/24' network to node1, node2 on your external gateway, so that the
897 public network can reply back.
899 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
900 and 10.0.2.0/24 in this example), will be announced dynamically.
906 VXLAN IPSEC Encryption
907 ~~~~~~~~~~~~~~~~~~~~~~
908 If you need to add encryption on top of VXLAN, it's possible to do so with
909 IPSEC through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
910 or 80 bytes (IPv6) to handle encryption.
912 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
913 + 50 (VXLAN) == 1500).
917 apt install strongswan
920 Add configuration in `/etc/ipsec.conf'. We only need to encrypt traffic from
921 the VXLAN UDP port '4789'.
925 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
927 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
930 rightsubnet=%dynamic[udp/4789]
937 leftsubnet=%dynamic[udp/4789]
943 Then generate a preshared key with
946 openssl rand -base64 128
949 and copy the key in `/etc/ipsec.secrets' so that the file content looks like:
952 : PSK <generatedbase64key>
955 You need to copy the PSK and the config on other nodes.