:pve-toplevel:
endif::manvolnum[]
-The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
-virtual networks (vnets) at datacenter level.
+The **S**oftware **D**efined **N**etwork (SDN) feature allows you to create
+virtual networks (VNets) at the datacenter level.
WARNING: SDN is currently an **experimental feature** in {pve}. This
-Documentation for it is also still under development, ask on our
+documentation for it is also still under development. Ask on our
xref:getting_help[mailing lists or in the forum] for questions and feedback.
+[[pvesdn_installation]]
Installation
------------
-To enable the experimental SDN integration, you need to install
-"libpve-network-perl" package
+To enable the experimental Software Defined Network (SDN) integration, you need
+to install the `libpve-network-perl` and `ifupdown2` packages on every node:
----
-apt install libpve-network-perl
+apt update
+apt install libpve-network-perl ifupdown2
----
-You need to have `ifupdown2` package installed on each node to manage local
-configuration reloading without reboot:
+NOTE: {pve} version 7 and above come installed with ifupdown2.
+
+After this, you need to add the following line to the end of the
+`/etc/network/interfaces` configuration file, so that the SDN configuration gets
+included and activated.
----
-apt install ifupdown2
+source /etc/network/interfaces.d/*
----
+
Basic Overview
--------------
-The {pve} SDN allows separation and fine grained control of Virtual Guests
-networks, using flexible software controlled configurations.
+The {pve} SDN allows for separation and fine-grained control of virtual guest
+networks, using flexible, software-controlled configurations.
-Separation consists of zones, a zone is it's own virtual separated area.
-A Zone can be used by one or more 'VNets'. A 'VNet' is virtual network in a
-zone. Normally it shows up as a common Linux bridge with either a VLAN or
-'VXLAN' tag, or using layer 3 routing for control.
-The 'VNets' are deployed locally on each node, after configuration was commited
-from the cluster wide datacenter level.
+Separation is managed through zones, where a zone is its own virtual separated
+network area. A 'VNet' is a type of a virtual network connected to a zone.
+Depending on which type or plugin the zone uses, it can behave differently and
+offer different features, advantages, and disadvantages. Normally, a 'VNet'
+appears as a common Linux bridge with either a VLAN or 'VXLAN' tag, however,
+some can also use layer 3 routing for control. 'VNets' are deployed locally on
+each node, after being configured from the cluster-wide datacenter SDN
+administration interface.
-Main configuration
-------------------
+Main Configuration
+~~~~~~~~~~~~~~~~~~
-The configuration is done at datacenter (cluster-wide) level, it will be saved
-in configuration files located in the shared configuration file system:
+Configuration is done at the datacenter (cluster-wide) level and is saved in
+files located in the shared configuration file system:
`/etc/pve/sdn`
-On the web-interface SDN feature have 4 main sections for the configuration
+On the web-interface, SDN features 3 main sections:
-* SDN: a overview of the SDN state
+* SDN: An overview of the SDN state
-* Zones: Create and manage the virtual separated network Zones
+* Zones: Create and manage the virtually separated network zones
-* VNets: The per-node building block to provide a Zone for VMs
+* VNets: Create virtual network bridges and manage subnets
-* Controller:
+In addition to this, the following options are offered:
+* Controller: For controlling layer 3 routing in complex setups
-SDN
-~~~
+* Subnets: Used to defined IP networks on VNets
-This is the main status panel. Here you can see deployment status of zones on
-different nodes.
+* IPAM: Enables the use of external tools for IP address management (guest
+ IPs)
-There is an 'Apply' button, to push and reload local configuration on all
-cluster nodes nodes.
+* DNS: Define a DNS server API for registering virtual guests' hostname and IP
+ addresses
+[[pvesdn_config_main_sdn]]
-Zones
-~~~~~
+SDN
+~~~
-A zone will define a virtually separated network.
+This is the main status panel. Here you can see the deployment status of zones
+on different nodes.
-It can use different technologies for separation:
+The 'Apply' button is used to push and reload local configuration on all cluster
+nodes.
-* VLAN: Virtual LANs are the classic method to sub-divide a LAN
-* QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
+[[pvesdn_local_deployment_monitoring]]
+Local Deployment Monitoring
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
-* VXLAN: (layer2 vxlan)
+After applying the configuration through the main SDN panel,
+the local network configuration is generated locally on each node in
+the file `/etc/network/interfaces.d/sdn`, and reloaded with ifupdown2.
-* bgp-evpn: vxlan using layer3 border gateway protocol routing
+You can monitor the status of local zones and VNets through the main tree.
-You can restrict a zone to specific nodes.
-It's also possible to add permissions on a zone, to restrict user to use only a
-specific zone and only the VNets in that zone
+[[pvesdn_config_zone]]
+Zones
+-----
-VNets
-~~~~~
+A zone defines a virtually separated network. Zones can be restricted to
+specific nodes and assigned permissions, in order to restrict users to a certain
+zone and its contained VNets.
-A `VNet` is in its basic form just a Linux bridge that will be deployed locally
-on the node and used for Virtual Machine communication.
+Different technologies can be used for separation:
-VNet properties are:
+* VLAN: Virtual LANs are the classic method of subdividing a LAN
-* ID: a 8 characters ID to name and identify a VNet
+* QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
-* Alias: Optional longer name, if the ID isn't enough
+* VXLAN: Layer2 VXLAN
-* Zone: The associated zone for this VNet
+* Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
-* Tag: The unique VLAN or VXLAN id
+* EVPN (BGP EVPN): VXLAN using layer 3 border gateway protocol (BGP) routing
-* IPv4: an anycast IPv4 address, it will be configured on the underlying bridge
- on each node part of the Zone. It's only useful for `bgp-evpn` routing.
+Common options
+~~~~~~~~~~~~~~
-* IPv6: an anycast IPv6 address, it will be configured on the underlying bridge
- on each node part of the Zone. It's only useful for `bgp-evpn` routing.
+The following options are available for all zone types:
+nodes:: The nodes which the zone and associated VNets should be deployed on
-Controllers
-~~~~~~~~~~~
+ipam:: Optional. Use an IP Address Management (IPAM) tool to manage IPs in the
+ zone.
-Some zone types (currently only the `bgp-evpn` plugin) need an external
-controller to manage the VNet control-plane.
+dns:: Optional. DNS API server.
+reversedns:: Optional. Reverse DNS API server.
-Zones Plugins
--------------
+dnszone:: Optional. DNS domain name. Used to register hostnames, such as
+ `<hostname>.<domain>`. The DNS zone must already exist on the DNS server.
-Common options
-~~~~~~~~~~~~~~
-nodes:: deploy and allow to use a VNets configured for this Zone only on
-these nodes.
+[[pvesdn_zone_plugin_simple]]
+Simple Zones
+~~~~~~~~~~~~
+This is the simplest plugin. It will create an isolated VNet bridge.
+This bridge is not linked to a physical interface, and VM traffic is only
+local between the node(s).
+It can also be used in NAT or routed setups.
+[[pvesdn_zone_plugin_vlan]]
VLAN Zones
~~~~~~~~~~
-This is the simplest plugin, it will reuse an existing local Linux or OVS
-bridge, and manage VLANs on it.
-The benefit of using SDN module, is that you can create different zones with
-specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
+This plugin reuses an existing local Linux or OVS bridge, and manages the VLANs
+on it. The benefit of using the SDN module is that you can create different
+zones with specific VNet VLAN tags, and restrict virtual machines to separated
+zones.
Specific `VLAN` configuration options:
-bridge:: Reuse this local VLAN-aware bridge, or OVS interface, already
-configured on *each* local node.
+bridge:: Reuse this local bridge or OVS switch, already configured on *each*
+ local node.
+[[pvesdn_zone_plugin_qinq]]
QinQ Zones
~~~~~~~~~~
-QinQ is stacked VLAN. The first VLAN tag defined for the zone
-(so called 'service-vlan'), and the second VLAN tag defined for the vnets
+QinQ also known as VLAN stacking, wherein the first VLAN tag is defined for the
+zone (the 'service-vlan'), and the second VLAN tag is defined for the
+VNets.
+
+NOTE: Your physical network switches must support stacked VLANs for this
+configuration!
-NOTE: Your physical network switchs must support stacked VLANs!
+Below are the configuration options specific to QinQ:
-Specific QinQ configuration options:
+bridge:: A local, VLAN-aware bridge that is already configured on each local
+ node
-bridge:: a local VLAN-aware bridge already configured on each local node
-service vlan:: he main VLAN tag of this zone
-mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
-For example, you reduce the MTU to `1496` if you physical interface MTU is
-`1500`.
+service vlan:: The main VLAN tag of this zone
+service vlan protocol:: Allows you to choose between an 802.1q (default) or
+ 802.1ad service VLAN type.
+
+mtu:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
+ For example, you must reduce the MTU to `1496` if you physical interface MTU is
+ `1500`.
+
+[[pvesdn_zone_plugin_vxlan]]
VXLAN Zones
~~~~~~~~~~~
-The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
-network (named underlay). It encapsulate layer 2 Ethernet frames within layer
+The VXLAN plugin establishes a tunnel (overlay) on top of an existing
+network (underlay). This encapsulates layer 2 Ethernet frames within layer
4 UDP datagrams, using `4789` as the default destination port. You can, for
example, create a private IPv4 VXLAN network on top of public internet network
nodes.
-This is a layer2 tunnel only, no routing between different VNets is possible.
-Each VNet will have use specific VXLAN id from the range (1 - 16777215).
+This is a layer 2 tunnel only, so no routing between different VNets is
+possible.
+
+Each VNet will have a specific VXLAN ID in the range 1 - 16777215.
Specific EVPN configuration options:
-peers address list:: a list of IPs from all nodes where you want to communicate (can also be external nodes)
-mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes lower than the outgoing physical interface.
+peers address list:: A list of IP addresses from each node through which you
+ want to communicate. Can also be external nodes.
+
+mtu:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ lower than the outgoing physical interface.
+[[pvesdn_zone_plugin_evpn]]
EVPN Zones
~~~~~~~~~~
-This is the most complex of all supported plugins.
+This is the most complex of all the supported plugins.
-BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
-have an anycast IP-address and or MAC-address. The bridge IP is the same on each
-node, with this a virtual guest can use that address as gateway.
+BGP-EVPN allows you to create a routable layer 3 network. The VNet of EVPN can
+have an anycast IP address and/or MAC address. The bridge IP is the same on each
+node, meaning a virtual guest can use this address as gateway.
Routing can work across VNets from different zones through a VRF (Virtual
Routing and Forwarding) interface.
-Specific EVPN configuration options:
+The configuration options specific to EVPN are as follows:
+
+VRF VXLAN tag:: This is a VXLAN-ID used for routing interconnect between VNets.
+ It must be different than the VXLAN-ID of the VNets.
+
+controller:: An EVPN-controller must to be defined first (see controller plugins
+ section).
+
+VNet MAC address:: A unique, anycast MAC address for all VNets in this zone.
+ Will be auto-generated if not defined.
+
+Exit Nodes:: Optional. This is used if you want to define some {pve} nodes as
+ exit gateways from the EVPN network, through the real network. The configured
+ nodes will announce a default route in the EVPN network.
+
+Primary Exit Node:: Optional. If you use multiple exit nodes, this forces
+ traffic to a primary exit node, instead of load-balancing on all nodes. This
+ is required if you want to use SNAT or if your upstream router doesn't support
+ ECMP.
+
+Exit Nodes local routing:: Optional. This is a special option if you need to
+ reach a VM/CT service from an exit node. (By default, the exit nodes only
+ allow forwarding traffic between real network and EVPN network).
+
+Advertise Subnets:: Optional. If you have silent VMs/CTs (for example, if you
+ have multiple IPs and the anycast gateway doesn't see traffic from theses IPs,
+ the IP addresses won't be able to be reach inside the EVPN network). This
+ option will announce the full subnet in the EVPN network in this case.
-VRF VXLAN Tag:: This is a vxlan-id used for routing interconnect between vnets,
-it must be different than VXLAN-id of VNets
+Disable Arp-Nd Suppression:: Optional. Don't suppress ARP or ND packets.
+ This is required if you use floating IPs in your guest VMs
+ (IP are MAC addresses are being moved between systems).
-controller:: an EVPN-controller need to be defined first (see controller
-plugins section)
+Route-target import:: Optional. Allows you to import a list of external EVPN
+ route targets. Used for cross-DC or different EVPN network interconnects.
-mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
-lower than the outgoing physical interface.
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ less than the maximal MTU of the outgoing physical interface.
-Controllers Plugins
--------------------
+[[pvesdn_config_vnet]]
+VNets
+-----
+
+A `VNet` is, in its basic form, a Linux bridge that will be deployed locally on
+the node and used for virtual machine communication.
+
+The VNet configuration properties are:
+
+ID:: An 8 character ID to name and identify a VNet
+
+Alias:: Optional longer name, if the ID isn't enough
+
+Zone:: The associated zone for this VNet
+
+Tag:: The unique VLAN or VXLAN ID
+
+VLAN Aware:: Enable adding an extra VLAN tag in the virtual machine or
+container's vNIC configuration, to allow the guest OS to manage the VLAN's tag.
+
+[[pvesdn_config_subnet]]
+Subnets
+~~~~~~~~
+
+A subnetwork (subnet) allows you to define a specific IP network
+(IPv4 or IPv6). For each VNet, you can define one or more subnets.
+
+A subnet can be used to:
+
+* Restrict the IP addresses you can define on a specific VNet
+* Assign routes/gateways on a VNet in layer 3 zones
+* Enable SNAT on a VNet in layer 3 zones
+* Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
+* DNS registration through DNS plugins
+
+If an IPAM server is associated with the subnet zone, the subnet prefix will be
+automatically registered in the IPAM.
+
+Subnet properties are:
+
+ID:: A CIDR network address, for example 10.0.0.0/8
+
+Gateway:: The IP address of the network's default gateway. On layer 3 zones
+ (Simple/EVPN plugins), it will be deployed on the VNet.
+SNAT:: Optional. Enable SNAT for layer 3 zones (Simple/EVPN plugins), for this
+ subnet. The subnet's source IP will be NATted to server's outgoing interface/IP.
+ On EVPN zones, this is only done on EVPN gateway-nodes.
+
+Dnszoneprefix:: Optional. Add a prefix to the domain registration, like
+<hostname>.prefix.<domain>
+
+[[pvesdn_config_controllers]]
+Controllers
+-----------
+
+Some zone types need an external controller to manage the VNet control-plane.
+Currently this is only required for the `bgp-evpn` zone plugin.
+
+[[pvesdn_controller_plugin_evpn]]
EVPN Controller
~~~~~~~~~~~~~~~
You may need to install it on each node where you want to deploy EVPN zones.
----
-apt install frr
+apt install frr frr-pythontools
----
Configuration options:
-asn:: a unique BGP ASN number. It's highly recommended to use private ASN
-number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
-breaking, or get broken, by global routing by mistake.
+asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
+ breaking global routing by mistake.
-peers:: an ip list of all nodes where you want to communicate (could be also
-external nodes or route reflectors servers)
+peers:: An IP list of all nodes where you want to communicate for the EVPN
+ (could also be external nodes or route reflectors servers)
-Additionally, if you want to route traffic from a SDN BGP-EVPN network to
-external world:
-gateway-nodes:: The proxmox nodes from where the bgp-evpn traffic will exit to
-external through the nodes default gateway
+[[pvesdn_controller_plugin_BGP]]
+BGP Controller
+~~~~~~~~~~~~~~~
-If you want that gateway nodes don't use the default gateway, but, for example,
-sent traffic to external BGP routers
+The BGP controller is not used directly by a zone.
+You can use it to configure FRR to manage BGP peers.
-gateway-external-peers:: 192.168.0.253,192.168.0.254
+For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
+Configuration options:
-Local Deployment Monitoring
----------------------------
+node:: The node of this BGP controller
-After applying the configuration through the main SDN web-interface panel,
-the local network configuration is generated locally on each node in
-`/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
+asn:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
+ you could break global routing by mistake.
+
+peers:: A list of peer IP addresses you want to communicate with using the
+ underlying BGP network.
+
+ebgp:: If your peer's remote-AS is different, this enables EBGP.
+
+loopback:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
+
+ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
+ not directly connected or they use loopback.
+
+bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
+
+[[pvesdn_config_ipam]]
+IPAMs
+-----
+
+IPAM (IP Address Management) tools are used to manage/assign the IP addresses of
+guests on the network. It can be used to find free IP addresses when you create
+a VM/CT for example (not yet implemented).
-You can monitor the status of local zones and vnets through the main tree.
+An IPAM can be associated with one or more zones, to provide IP addresses
+for all subnets defined in those zones.
+[[pvesdn_ipam_plugin_pveipam]]
+{pve} IPAM Plugin
+~~~~~~~~~~~~~~~~~
+This is the default internal IPAM for your {pve} cluster, if you don't have
+external IPAM software.
+
+[[pvesdn_ipam_plugin_phpipam]]
+phpIPAM Plugin
+~~~~~~~~~~~~~~
+https://phpipam.net/
+
+You need to create an application in phpIPAM and add an API token with admin
+privileges.
+
+The phpIPAM configuration properties are:
+
+url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
+
+token:: An API access token
+
+section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
+ installations use `sectionid=1` for customers.
+
+[[pvesdn_ipam_plugin_netbox]]
+NetBox IPAM Plugin
+~~~~~~~~~~~~~~~~~~
+
+NetBox is an IP address management (IPAM) and datacenter infrastructure
+management (DCIM) tool. See the source code repository for details:
+https://github.com/netbox-community/netbox
+
+You need to create an API token in NetBox to use it:
+https://netbox.readthedocs.io/en/stable/api/authentication
+
+The NetBox configuration properties are:
+
+url:: The REST API endpoint: `http://yournetbox.domain.com/api`
+
+token:: An API access token
+
+[[pvesdn_config_dns]]
+DNS
+---
+
+The DNS plugin in {pve} SDN is used to define a DNS API server for registration
+of your hostname and IP address. A DNS configuration is associated with one or
+more zones, to provide DNS registration for all the subnet IPs configured for
+a zone.
+
+[[pvesdn_dns_plugin_powerdns]]
+PowerDNS Plugin
+~~~~~~~~~~~~~~~
+https://doc.powerdns.com/authoritative/http-api/index.html
+
+You need to enable the web server and the API in your PowerDNS config:
+
+----
+api=yes
+api-key=arandomgeneratedstring
+webserver=yes
+webserver-port=8081
+----
+
+The PowerDNS configuration options are:
+
+url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
+
+key:: An API access key
+
+ttl:: The default TTL for records
+
+
+Examples
+--------
+
+[[pvesdn_setup_example_vlan]]
VLAN Setup Example
-------------------
+~~~~~~~~~~~~~~~~~~
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
+TIP: While we show plaintext configuration content here, almost everything
+should be configurable using the web-interface only.
Node1: /etc/network/interfaces
----
Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
-`myvlanzone' as it's zone.
+`myvlanzone' as its zone.
----
id: myvnet1
----
Apply the configuration through the main SDN panel, to create VNets locally on
-each nodes.
+each node.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
Use the following network configuration for this VM:
address 10.0.3.100/24
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
+Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
`myvnet1' as vm1.
Use the following network configuration for this VM:
address 10.0.3.101/24
----
-Then, you should be able to ping between both VMs over that network.
+Following this, you should be able to ping between both VMs over that network.
-QinQ setup example
-------------------
+[[pvesdn_setup_example_qinq]]
+QinQ Setup Example
+~~~~~~~~~~~~~~~~~~
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
+TIP: While we show plaintext configuration content here, almost everything
+should be configurable using the web-interface only.
Node1: /etc/network/interfaces
source /etc/network/interfaces.d/*
----
-Create an QinQ zone named `qinqzone1' with service VLAN 20
+Create a QinQ zone named `qinqzone1' with service VLAN 20
----
id: qinqzone1
service vlan: 30
----
-Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
+Create a VNet named `myvnet1' with customer VLAN-ID 100 on the previously
created `qinqzone1' zone.
----
tag: 100
----
-Create a `myvnet2' with customer VLAN-id 100 on the previously created
+Create a `myvnet2' with customer VLAN-ID 100 on the previously created
`qinqzone2' zone.
----
id: myvnet2
-zone: qinqzone1
+zone: qinqzone2
tag: 100
----
Apply the configuration on the main SDN web-interface panel to create VNets
locally on each nodes.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
Use the following network configuration for this VM:
address 10.0.3.100/24
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
+Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
`myvnet1' as vm1.
Use the following network configuration for this VM:
address 10.0.3.101/24
----
-Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
+Create a third virtual machine (vm3) on node1, with a vNIC on the other VNet
`myvnet2'.
Use the following network configuration for this VM:
address 10.0.3.102/24
----
-Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
+Create another virtual machine (vm4) on node2, with a vNIC on the same VNet
`myvnet2' as vm3.
Use the following network configuration for this VM:
address 10.0.3.103/24
----
-Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
-between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
-or 'vm4', as they are on a different zone with different service-vlan.
+Then, you should be able to ping between the VMs 'vm1' and 'vm2', as well as
+between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
+'vm3' or 'vm4', as they are on a different zone with a different service-vlan.
+[[pvesdn_setup_example_vxlan]]
VXLAN Setup Example
--------------------
+~~~~~~~~~~~~~~~~~~~
+
+TIP: While we show plaintext configuration content here, almost everything
+is configurable through the web-interface.
node1: /etc/network/interfaces
source /etc/network/interfaces.d/*
----
-Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
+Create a VXLAN zone named `myvxlanzone', using a lower MTU to ensure the extra
50 bytes of the VXLAN header can fit. Add all previously configured IPs from
-the nodes as peer address list.
+the nodes to the peer address list.
----
id: myvxlanzone
Apply the configuration on the main SDN web-interface panel to create VNets
locally on each nodes.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
-Use the following network configuration for this VM, note the lower MTU here.
+Use the following network configuration for this VM (note the lower MTU).
----
auto eth0
mtu 1450
----
-Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
+Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet
`myvnet1' as vm1.
Use the following network configuration for this VM:
Then, you should be able to ping between between 'vm1' and 'vm2'.
-
-EVPN setup example
-------------------
+[[pvesdn_setup_example_evpn]]
+EVPN Setup Example
+~~~~~~~~~~~~~~~~~~
node1: /etc/network/interfaces
source /etc/network/interfaces.d/*
----
-Create a EVPN controller, using a private ASN number and above node addreesses
-as peers. Define 'node1' and 'node2' as gateway nodes.
+Create an EVPN controller, using a private ASN number and the above node
+addresses as peers.
----
id: myevpnctl
asn: 65000
peers: 192.168.0.1,192.168.0.2,192.168.0.3
-gateway nodes: node1,node2
----
-Create an EVPN zone named `myevpnzone' using the previously created
-EVPN-controller.
+Create an EVPN zone named `myevpnzone', using the previously created
+EVPN-controller. Define 'node1' and 'node2' as exit nodes.
----
id: myevpnzone
vrf vxlan tag: 10000
controller: myevpnctl
mtu: 1450
+vnet mac address: 32:F4:05:FE:6C:0A
+exitnodes: node1,node2
----
-Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone', a IPv4
-CIDR network and a random MAC address.
-
+Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
----
id: myvnet1
zone: myevpnzone
tag: 11000
-ipv4: 10.0.1.1/24
-mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
+----
+
+Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on `myvnet1`.
+
+----
+subnet: 10.0.1.0/24
+gateway: 10.0.1.1
----
Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
-different IPv4 CIDR network and a different random MAC address than `myvnet1'.
+different IPv4 CIDR network.
----
id: myvnet2
zone: myevpnzone
tag: 12000
-ipv4: 10.0.2.1/24
-mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet
----
-Apply the configuration on the main SDN web-interface panel to create VNets
-locally on each nodes and generate the FRR config.
+Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
+
+----
+subnet: 10.0.2.0/24
+gateway: 10.0.2.1
+----
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Apply the configuration from the main SDN web-interface panel to create VNets
+locally on each node and generate the FRR config.
+
+Create a Debian-based virtual machine (vm1) on node1, with a vNIC on `myvnet1'.
Use the following network configuration for this VM:
mtu 1450
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
+Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet
`myvnet2'.
Use the following network configuration for this VM:
auto eth0
iface eth0 inet static
address 10.0.2.100/24
- gateway 10.0.2.1 #this is the ip of the vnet2
+ gateway 10.0.2.1 #this is the ip of the myvnet2
mtu 1450
----
Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
-will go to the configured 'myvnet2' gateway, then will be routed to gateway
+will go to the configured 'myvnet2' gateway, then will be routed to the exit
nodes ('node1' or 'node2') and from there it will leave those nodes over the
default gateway configured on node1 or node2.
-NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
-'10.0.2.0/24' network to node1, node2 on your external gateway, so that the
-public network can reply back.
+NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
+networks to node1 and node2 on your external gateway, so that the public network
+can reply back.
If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
and 10.0.2.0/24 in this example), will be announced dynamically.
+
+
+Notes
+-----
+
+VXLAN IPSEC Encryption
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you need to add encryption on top of a VXLAN, it's possible to do so with
+IPSEC, through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
+or 80 bytes (IPv6) to handle encryption.
+
+So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
++ 50 (VXLAN) == 1500).
+
+.Install strongswan
+----
+apt install strongswan
+----
+
+Add configuration to `/etc/ipsec.conf'. We only need to encrypt traffic from
+the VXLAN UDP port '4789'.
+
+----
+conn %default
+ ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
+ esp=aes256-sha1!
+ leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
+
+conn output
+ rightsubnet=%dynamic[udp/4789]
+ right=%any
+ type=transport
+ authby=psk
+ auto=route
+
+conn input
+ leftsubnet=%dynamic[udp/4789]
+ type=transport
+ authby=psk
+ auto=route
+----
+
+Then generate a pre-shared key with:
+
+----
+openssl rand -base64 128
+----
+
+and add the key to `/etc/ipsec.secrets', so that the file contents looks like:
+
+----
+: PSK <generatedbase64key>
+----
+
+You need to copy the PSK and the configuration onto the other nodes.
projects / pve-docs.git / blobdiff