2 Software-Defined Network
3 ========================
8 The **S**oftware-**D**efined **N**etwork (SDN) feature in {pve} enables the
9 creation of virtual zones and networks (VNets). This functionality simplifies
10 advanced networking configurations and multitenancy setup.
16 The {pve} SDN allows for separation and fine-grained control of virtual guest
17 networks, using flexible, software-controlled configurations.
19 Separation is managed through *zones*, virtual networks (*VNets*), and
20 *subnets*. A zone is its own virtually separated network area. A VNet is a
21 virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
23 Depending on the type of the zone, the network behaves differently and offers
24 specific features, advantages, and limitations.
26 Use cases for SDN range from an isolated private network on each individual node
27 to complex overlay networks across multiple PVE clusters on different locations.
29 After configuring an VNet in the cluster-wide datacenter SDN administration
30 interface, it is available as a common Linux bridge, locally on each node, to be
31 assigned to VMs and Containers.
34 [[pvesdn_support_status]]
41 The {pve} SDN stack has been available as an experimental feature since 2019 and
42 has been continuously improved and tested by many developers and users.
43 With its integration into the web interface in {pve} 6.2, a significant
44 milestone towards broader integration was achieved.
45 During the {pve} 7 release cycle, numerous improvements and features were added.
46 Based on user feedback, it became apparent that the fundamental design choices
47 and their implementation were quite sound and stable. Consequently, labeling it
48 as `experimental' did not do justice to the state of the SDN stack.
49 For {pve} 8, a decision was made to lay the groundwork for full integration of
50 the SDN feature by elevating the management of networks and interfaces to a core
51 component in the {pve} access control stack.
52 In {pve} 8.1, two major milestones were achieved: firstly, DHCP integration was
53 added to the IP address management (IPAM) feature, and secondly, the SDN
54 integration is now installed by default.
59 The current support status for the various layers of our SDN installation is as
62 - Core SDN, which includes VNet management and its integration with the {pve}
63 stack, is fully supported.
64 - IPAM, including DHCP management for virtual guests, is in tech preview.
65 - Complex routing via FRRouting and controller integration are in tech preview.
67 [[pvesdn_installation]]
74 Since {pve} 8.1 the core Software-Defined Network (SDN) packages are installed
77 If you upgrade from an older version, you need to install the
78 `libpve-network-perl` package on every node:
82 apt install libpve-network-perl
85 NOTE: {pve} version 7.0 and above have the `ifupdown2` package installed by
86 default. If you originally installed your system with an older version, you need
87 to explicitly install the `ifupdown2` package.
89 After installation, you need to ensure that the following line is present at the
90 end of the `/etc/network/interfaces` configuration file on all nodes, so that
91 the SDN configuration gets included and activated.
94 source /etc/network/interfaces.d/*
97 [[pvesdn_install_dhcp_ipam]]
101 The DHCP integration into the built-in 'PVE' IP Address Management stack
102 currently uses `dnsmasq` for giving out DHCP leases. This is currently opt-in.
104 To use that feature you need to install the `dnsmasq` package on every node:
109 # disable default instance
110 systemctl disable --now dnsmasq
113 [[pvesdn_install_frrouting]]
117 The {pve} SDN stack uses the https://frrouting.org/[FRRouting] project for
118 advanced setups. This is currently opt-in.
120 To use the SDN routing integration you need to install the `frr-pythontools`
121 package on all nodes:
125 apt install frr-pythontools
128 [[pvesdn_main_configuration]]
129 Configuration Overview
130 ----------------------
132 Configuration is done at the web UI at datacenter level, separated into the
135 * SDN:: Here you get an overview of the current active SDN state, and you can
136 apply all pending changes to the whole cluster.
138 * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
141 * xref:pvesdn_config_vnet[VNets] VNets: Create virtual network bridges and
144 The Options category allows adding and managing additional services to be used
147 * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
150 * DHCP: Define a DHCP server for a zone that automatically allocates IPs for
151 guests in the IPAM and leases them to the guests via DHCP.
153 * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
156 * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
157 virtual guests' hostname and IP addresses
159 [[pvesdn_tech_and_config_overview]]
160 Technology & Configuration
161 --------------------------
163 The {pve} Software-Defined Network implementation uses standard Linux networking
164 as much as possible. The reason for this is that modern Linux networking
165 provides almost all needs for a feature full SDN implementation and avoids adding
166 external dependencies and reduces the overall amount of components that can
169 The {pve} SDN configurations are located in `/etc/pve/sdn`, which is shared with
170 all other cluster nodes through the {pve} xref:chapter_pmxcfs[configuration file system].
171 Those configurations get translated to the respective configuration formats of
172 the tools that manage the underlying network stack (for example `ifupdown2` or
175 New changes are not immediately applied but recorded as pending first. You can
176 then apply a set of different changes all at once in the main 'SDN' overview
177 panel on the web interface. This system allows to roll-out various changes as
180 The SDN tracks the rolled-out state through the '.running-config' and '.version'
181 files located in '/etc/pve/sdn'.
183 // TODO: extend implementation and technology details.
185 [[pvesdn_config_zone]]
189 A zone defines a virtually separated network. Zones are restricted to
190 specific nodes and assigned permissions, in order to restrict users to a certain
191 zone and its contained VNets.
193 Different technologies can be used for separation:
195 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
197 * VLAN: Virtual LANs are the classic method of subdividing a LAN
199 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
201 * VXLAN: Layer 2 VXLAN network via a UDP tunnel
203 * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
206 [[pvesdn_config_common_options]]
210 The following options are available for all zone types:
212 Nodes:: The nodes which the zone and associated VNets should be deployed on.
214 IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
215 zone. Optional, defaults to `pve`.
217 DNS:: DNS API server. Optional.
219 ReverseDNS:: Reverse DNS API server. Optional.
221 DNSZone:: DNS domain name. Used to register hostnames, such as
222 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
225 [[pvesdn_zone_plugin_simple]]
229 This is the simplest plugin. It will create an isolated VNet bridge. This
230 bridge is not linked to a physical interface, and VM traffic is only local on
232 It can be used in NAT or routed setups.
235 [[pvesdn_zone_plugin_vlan]]
239 The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
240 node's physical interface. It uses VLAN tagging defined in the VNet to isolate
241 the network segments. This allows connectivity of VMs between different nodes.
243 VLAN zone configuration options:
245 Bridge:: The local bridge or OVS switch, already configured on *each* node that
246 allows node-to-node connection.
249 [[pvesdn_zone_plugin_qinq]]
253 QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
254 isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
255 whereas the inner VLAN tag is defined by the VNet.
257 NOTE: Your physical network switches must support stacked VLANs for this
260 QinQ zone configuration options:
262 Bridge:: A local, VLAN-aware bridge that is already configured on each local
265 Service VLAN:: The main VLAN tag of this zone
267 Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
268 802.1ad service VLAN type.
270 MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
271 For example, you must reduce the MTU to `1496` if you physical interface MTU is
275 [[pvesdn_zone_plugin_vxlan]]
279 The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
280 (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
281 datagrams using the default destination port `4789`.
283 You have to configure the underlay network yourself to enable UDP connectivity
286 You can, for example, create a VXLAN overlay network on top of public internet,
287 appearing to the VMs as if they share the same local Layer 2 network.
289 WARNING: VXLAN on its own does does not provide any encryption. When joining
290 multiple sites via VXLAN, make sure to establish a secure connection between
291 the site, for example by using a site-to-site VPN.
293 VXLAN zone configuration options:
295 Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
296 can be external nodes reachable at this IP address.
297 All nodes in the cluster need to be mentioned here.
299 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
300 lower than the outgoing physical interface.
303 [[pvesdn_zone_plugin_evpn]]
307 The EVPN zone creates a routable Layer 3 network, capable of spanning across
308 multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
309 the routing protocol.
311 The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
312 IP is the same on each node, meaning a virtual guest can use this address as
315 Routing can work across VNets from different zones through a VRF (Virtual
316 Routing and Forwarding) interface.
318 EVPN zone configuration options:
320 VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
321 It must be different than the VXLAN-ID of the VNets.
323 Controller:: The EVPN-controller to use for this zone. (See controller plugins
326 VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
327 zone. Will be auto-generated if not defined.
329 Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
330 network, through the real network. The configured nodes will announce a
331 default route in the EVPN network. Optional.
333 Primary Exit Node:: If you use multiple exit nodes, force traffic through this
334 primary exit node, instead of load-balancing on all nodes. Optional but
335 necessary if you want to use SNAT or if your upstream router doesn't support
338 Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
339 service from an exit node. (By default, the exit nodes only allow forwarding
340 traffic between real network and EVPN network). Optional.
342 Advertise Subnets:: Announce the full subnet in the EVPN network.
343 If you have silent VMs/CTs (for example, if you have multiple IPs and the
344 anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
345 able to be reached inside the EVPN network). Optional.
347 Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
348 packets. This is required if you use floating IPs in your VMs (IP and MAC
349 addresses are being moved between systems). Optional.
351 Route-target Import:: Allows you to import a list of external EVPN route
352 targets. Used for cross-DC or different EVPN network interconnects. Optional.
354 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
355 less than the maximal MTU of the outgoing physical interface. Optional,
359 [[pvesdn_config_vnet]]
363 After creating a virtual network (VNet) through the SDN GUI, a local network
364 interface with the same name is available on each node. To connect a guest to the
365 VNet, assign the interface to the guest and set the IP address accordingly.
367 Depending on the zone, these options have different meanings and are explained
368 in the respective zone section in this document.
370 WARNING: In the current state, some options may have no effect or won't work in
373 VNet configuration options:
375 ID:: An up to 8 character ID to identify a VNet
377 Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
379 Zone:: The associated zone for this VNet
381 Tag:: The unique VLAN or VXLAN ID
383 VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
387 [[pvesdn_config_subnet]]
391 A subnet define a specific IP range, described by the CIDR network address.
392 Each VNet, can have one or more subnets.
394 A subnet can be used to:
396 * Restrict the IP addresses you can define on a specific VNet
397 * Assign routes/gateways on a VNet in layer 3 zones
398 * Enable SNAT on a VNet in layer 3 zones
399 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
400 * DNS registration through DNS plugins
402 If an IPAM server is associated with the subnet zone, the subnet prefix will be
403 automatically registered in the IPAM.
405 Subnet configuration options:
407 ID:: A CIDR network address, for example 10.0.0.0/8
409 Gateway:: The IP address of the network's default gateway. On layer 3 zones
410 (Simple/EVPN plugins), it will be deployed on the VNet.
412 SNAT:: Enable Source NAT which allows VMs from inside a
413 VNet to connect to the outside network by forwarding the packets to the nodes
414 outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
417 DNS Zone Prefix:: Add a prefix to the domain registration, like
418 <hostname>.prefix.<domain> Optional.
421 [[pvesdn_config_controllers]]
425 Some zones implement a separated control and data plane that require an external
426 controller to manage the VNet's control plane.
428 Currently, only the `EVPN` zone requires an external controller.
431 [[pvesdn_controller_plugin_evpn]]
435 The `EVPN`, zone requires an external controller to manage the control plane.
436 The EVPN controller plugin configures the Free Range Routing (frr) router.
438 To enable the EVPN controller, you need to install frr on every node that shall
439 participate in the EVPN zone.
442 apt install frr frr-pythontools
445 EVPN controller configuration options:
447 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
448 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
449 breaking global routing by mistake.
451 Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
452 external nodes or route reflector servers)
455 [[pvesdn_controller_plugin_BGP]]
459 The BGP controller is not used directly by a zone.
460 You can use it to configure FRR to manage BGP peers.
462 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
463 It can also be used to export EVPN routes to an external BGP peer.
465 NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
468 BGP controller configuration options:
470 Node:: The node of this BGP controller
472 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
473 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
474 you could break global routing by mistake.
476 Peer:: A list of peer IP addresses you want to communicate with using the
477 underlying BGP network.
479 EBGP:: If your peer's remote-AS is different, this enables EBGP.
481 Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
484 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
485 not directly connected or they use loopback.
487 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
490 [[pvesdn_controller_plugin_ISIS]]
494 The ISIS controller is not used directly by a zone.
495 You can use it to configure FRR to export EVPN routes to an ISIS domain.
497 ISIS controller configuration options:
499 Node:: The node of this ISIS controller.
501 Domain:: A unique ISIS domain.
503 Network Entity Title:: A Unique ISIS network address that identifies this node.
505 Interfaces:: A list of physical interface(s) used by ISIS.
507 Loopback:: Use a loopback or dummy interface as the source of the EVPN network
511 [[pvesdn_config_ipam]]
515 IP Address Management (IPAM) tools manage the IP addresses of clients on the
516 network. SDN in {pve} uses IPAM for example to find free IP addresses for new
519 A single IPAM instance can be associated with one or more zones.
522 [[pvesdn_ipam_plugin_pveipam]]
526 The default built-in IPAM for your {pve} cluster.
528 You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in
529 the SDN section of the datacenter configuration. This UI can be used to create,
530 update and delete IP mappings. This is particularly convenient in conjunction
531 with the xref:pvesdn_config_dhcp[DHCP feature].
533 If you are using DHCP, you can use the IPAM panel to create or edit leases for
534 specific VMs, which enables you to change the IPs allocated via DHCP. When
535 editing an IP of a VM that is using DHCP you must make sure to force the guest
536 to acquire a new DHCP leases. This can usually be done by reloading the network
537 stack of the guest or rebooting it.
539 [[pvesdn_ipam_plugin_netbox]]
543 link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
544 Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
546 To integrate NetBox with {pve} SDN, create an API token in NetBox as described
547 here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
549 The NetBox configuration properties are:
551 URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
553 Token:: An API access token
556 [[pvesdn_ipam_plugin_phpipam]]
560 In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
561 an API token with admin privileges to the application.
563 The phpIPAM configuration properties are:
565 URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
567 Token:: An API access token
569 Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
570 installations use `sectionid=1` for customers.
573 [[pvesdn_config_dns]]
577 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
578 of your hostname and IP address. A DNS configuration is associated with one or
579 more zones, to provide DNS registration for all the subnet IPs configured for
582 [[pvesdn_dns_plugin_powerdns]]
585 https://doc.powerdns.com/authoritative/http-api/index.html
587 You need to enable the web server and the API in your PowerDNS config:
591 api-key=arandomgeneratedstring
596 The PowerDNS configuration options are:
598 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
600 key:: An API access key
602 ttl:: The default TTL for records
605 [[pvesdn_config_dhcp]]
609 The DHCP plugin in {pve} SDN can be used to automatically deploy a DHCP server
610 for a Zone. It provides DHCP for all Subnets in a Zone that have a DHCP range
611 configured. Currently the only available backend plugin for DHCP is the dnsmasq
614 The DHCP plugin works by allocating an IP in the IPAM plugin configured in the
615 Zone when adding a new network interface to a VM/CT. You can find more
616 information on how to configure an IPAM in the
617 xref:pvesdn_config_ipam[respective section of our documentation].
619 When the VM starts, a mapping for the MAC address and IP gets created in the DHCP
620 plugin of the zone. When the network interfaces is removed or the VM/CT are
621 destroyed, then the entry in the IPAM and the DHCP server are deleted as well.
623 NOTE: Some features (adding/editing/removing IP mappings) are currently only
624 available when using the xref:pvesdn_ipam_plugin_pveipam[PVE IPAM plugin].
630 You can enable automatic DHCP for a zone in the Web UI via the Zones panel and
631 enabling DHCP in the advanced options of a zone.
633 NOTE: Currently only Simple Zones have support for automatic DHCP
635 After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be
636 configured for the subnets in a Zone. In order to that, go to the Vnets panel and
637 select the Subnet for which you want to configure DHCP ranges. In the edit
638 dialogue you can configure DHCP ranges in the respective Tab. Alternatively you
639 can set DHCP ranges for a Subnet via the following CLI command:
642 pvesh set /cluster/sdn/vnets/<vnet>/subnets/<subnet>
643 -dhcp-range start-address=10.0.1.100,end-address=10.0.1.200
644 -dhcp-range start-address=10.0.2.100,end-address=10.0.2.200
647 You also need to have a gateway configured for the subnet - otherwise
648 automatic DHCP will not work.
650 The DHCP plugin will then allocate IPs in the IPAM only in the configured
653 Do not forget to follow the installation steps for the
654 xref:pvesdn_install_dhcp_ipam[dnsmasq DHCP plugin] as well.
661 Currently this is the only DHCP plugin and therefore the plugin that gets used
662 when you enable DHCP for a zone.
665 For installation see the xref:pvesdn_install_dhcp_ipam[DHCP IPAM] section.
668 The plugin will create a new systemd service for each zone that dnsmasq gets
669 deployed to. The name for the service is `dnsmasq@<zone>`. The lifecycle of this
670 service is managed by the DHCP plugin.
672 The plugin automatically generates the following configuration files in the
673 folder `/etc/dnsmasq.d/<zone>`:
676 This contains the default global configuration for a dnsmasq instance.
678 `10-<zone>-<subnet_cidr>.conf`::
679 This file configures specific options for a subnet, such as the DNS server that
680 should get configured via DHCP.
682 `10-<zone>-<subnet_cidr>.ranges.conf`::
683 This file configures the DHCP ranges for the dnsmasq instance.
686 This file contains the MAC-address and IP mappings from the IPAM plugin. In
687 order to override those mappings, please use the respective IPAM plugin rather
688 than editing this file, as it will get overwritten by the dnsmasq plugin.
690 You must not edit any of the above files, since they are managed by the DHCP
691 plugin. In order to customize the dnsmasq configuration you can create
692 additional files (e.g. `90-custom.conf`) in the configuration folder - they will
693 not get changed by the dnsmasq DHCP plugin.
695 Configuration files are read in order, so you can control the order of the
696 configuration directives by naming your custom configuration files appropriately.
698 DHCP leases are stored in the file `/var/lib/misc/dnsmasq.<zone>.leases`.
700 When using the PVE IPAM plugin, you can update, create and delete DHCP leases.
701 For more information please consult the documentation of
702 xref:pvesdn_ipam_plugin_pveipam[the PVE IPAM plugin]. Changing DHCP leases is
703 currently not supported for the other IPAM plugins.
705 [[pvesdn_setup_examples]]
709 This section presents multiple configuration examples tailored for common SDN
710 use cases. It aims to offer tangible implementations, providing additional
711 details to enhance comprehension of the available configuration options.
714 [[pvesdn_setup_example_simple]]
718 Simple zone networks create an isolated network for guests on a single host to
719 connect to each other.
721 TIP: connection between guests are possible if all guests reside on a same host
722 but cannot be reached on other nodes.
724 * Create a simple zone named `simple`.
725 * Add a VNet names `vnet1`.
726 * Create a Subnet with a gateway and the SNAT option enabled.
727 * This creates a network bridge `vnet1` on the node. Assign this bridge to the
728 guests that shall join the network and configure an IP address.
730 The network interface configuration in two VMs may look like this which allows
731 them to communicate via the 10.0.1.0/24 network.
735 iface ens19 inet static
741 iface ens19 inet static
746 [[pvesdn_setup_example_nat]]
750 If you want to allow outgoing connections for guests in the simple network zone
751 the simple zone offers a Source NAT (SNAT) option.
753 Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
754 Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
757 Subnet: 172.16.0.0/24
762 In the guests configure the static IP address inside the subnet's IP range.
764 The node itself will join this network with the Gateway IP '172.16.0.1' and
765 function as the NAT gateway for guests within the subnet range.
768 [[pvesdn_setup_example_vlan]]
772 When VMs on different nodes need to communicate through an isolated network, the
773 VLAN zone allows network level isolation using VLAN tags.
775 Create a VLAN zone named `myvlanzone`:
782 Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
791 Apply the configuration through the main SDN panel, to create VNets locally on
794 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
796 Use the following network configuration for this VM:
800 iface eth0 inet static
801 address 10.0.3.100/24
804 Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
807 Use the following network configuration for this VM:
811 iface eth0 inet static
812 address 10.0.3.101/24
815 Following this, you should be able to ping between both VMs using that network.
818 [[pvesdn_setup_example_qinq]]
823 This example configures two QinQ zones and adds two VMs to each zone to
824 demonstrate the additional layer of VLAN tags which allows the configuration of
827 A typical use case for this configuration is a hosting provider that provides an
828 isolated network to customers for VM communication but isolates the VMs from
831 Create a QinQ zone named `qinqzone1` with service VLAN 20
839 Create another QinQ zone named `qinqzone2` with service VLAN 30
846 Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
855 Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
863 Apply the configuration on the main SDN web interface panel to create VNets
864 locally on each node.
866 Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
867 interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
870 Inside the VM, configure the IP addresses of the interfaces, for example via
871 `/etc/network/interfaces`:
875 iface eth0 inet static
876 address 10.0.3.101/24
878 // TODO: systemd-network example
879 Configure all four VMs to have IP addresses from the '10.0.3.101' to
882 Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
883 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
884 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
887 [[pvesdn_setup_example_vxlan]]
891 The example assumes a cluster with three nodes, with the node IP addresses
892 192.168.0.1, 192.168.0.2 and 192.168.0.3.
894 Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
895 peer address list. Use the default MTU of 1450 or configure accordingly.
899 Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
902 Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
911 Apply the configuration on the main SDN web interface panel to create VNets
912 locally on each nodes.
914 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
916 Use the following network configuration for this VM (note the lower MTU).
920 iface eth0 inet static
921 address 10.0.3.100/24
925 Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
928 Use the following network configuration for this VM:
932 iface eth0 inet static
933 address 10.0.3.101/24
937 Then, you should be able to ping between between 'vm1' and 'vm2'.
940 [[pvesdn_setup_example_evpn]]
944 The example assumes a cluster with three nodes (node1, node2, node3) with IP
945 addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
947 Create an EVPN controller, using a private ASN number and the above node
953 Peers: 192.168.0.1,192.168.0.2,192.168.0.3
956 Create an EVPN zone named `myevpnzone`, assign the previously created
957 EVPN-controller and define 'node1' and 'node2' as exit nodes.
962 Controller: myevpnctl
964 VNet MAC Address: 32:F4:05:FE:6C:0A
965 Exit Nodes: node1,node2
968 Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
976 Create a subnet on `myvnet1`:
983 Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
991 Create a different subnet on `myvnet2``:
998 Apply the configuration from the main SDN web interface panel to create VNets
999 locally on each node and generate the FRR configuration.
1001 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
1003 Use the following network configuration for 'vm1':
1007 iface eth0 inet static
1008 address 10.0.1.100/24
1013 Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
1016 Use the following network configuration for 'vm2':
1020 iface eth0 inet static
1021 address 10.0.2.100/24
1027 Now you should be able to ping vm2 from vm1, and vm1 from vm2.
1029 If you ping an external IP from 'vm2' on the non-gateway node3, the packet
1030 will go to the configured 'myvnet2' gateway, then will be routed to the exit
1031 nodes ('node1' or 'node2') and from there it will leave those nodes over the
1032 default gateway configured on node1 or node2.
1034 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
1035 networks to node1 and node2 on your external gateway, so that the public network
1038 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
1039 and 10.0.2.0/24 in this example), will be announced dynamically.
1046 Multiple EVPN Exit Nodes
1047 ~~~~~~~~~~~~~~~~~~~~~~~~
1049 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
1050 Reverse Path Filter) option, because packets can arrive at one node but go out
1053 Add the following to `/etc/sysctl.conf`:
1056 net.ipv4.conf.default.rp_filter=0
1057 net.ipv4.conf.all.rp_filter=0
1060 VXLAN IPSEC Encryption
1061 ~~~~~~~~~~~~~~~~~~~~~~
1063 To add IPSEC encryption on top of a VXLAN, this example shows how to use
1066 You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
1067 IPv6 to handle encryption.
1069 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
1070 + 50 (VXLAN) == 1500).
1072 Install strongswan on the host.
1075 apt install strongswan
1078 Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
1079 the VXLAN UDP port '4789'.
1083 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
1085 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
1088 rightsubnet=%dynamic[udp/4789]
1095 leftsubnet=%dynamic[udp/4789]
1101 Generate a pre-shared key with:
1104 openssl rand -base64 128
1107 and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
1110 : PSK <generatedbase64key>
1113 Copy the PSK and the configuration to all nodes participating in the VXLAN network.
projects / pve-docs.git / blob