:pve-toplevel:
endif::manvolnum[]
-The SDN feature allow to create virtual networks (vnets)
-at datacenter level.
+The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
+virtual networks (vnets) at datacenter level.
-To enable SDN feature, you need to install "libpve-network-perl" package
+WARNING: SDN is currently an **experimental feature** in {pve}. This
+Documentation for it is also still under development, ask on our
+xref:getting_help[mailing lists or in the forum] for questions and feedback.
-----
-apt install libpve-network-perl
-----
-A vnet is a bridge with a vlan or vxlan tag.
+[[pvesdn_installation]]
+Installation
+------------
+
+To enable the experimental SDN integration, you need to install the
+`libpve-network-perl` and `ifupdown2` package on every node:
-The vnets are deployed locally on each node after configuration
-commit at datacenter level.
+----
+apt update
+apt install libpve-network-perl ifupdown2
+----
-You need to have "ifupdown2" package installed on each node to manage local
-configuration reloading.
+After that you need to add the following line:
----
-apt install ifupdown2
+source /etc/network/interfaces.d/*
----
+at the end of the `/etc/network/interfaces` configuration file, so that the SDN
+config gets included and activated.
+
+
+Basic Overview
+--------------
+
+The {pve} SDN allows separation and fine grained control of Virtual Guests
+networks, using flexible software controlled configurations.
+
+Separation consists of zones, a zone is it's own virtual separated network area.
+A 'VNet' is a type of a virtual network connected to a zone. Depending on which
+type or plugin the zone uses it can behave differently and offer different
+features, advantages or disadvantages.
+Normally a 'VNet' shows up as a common Linux bridge with either a VLAN or
+'VXLAN' tag, but some can also use layer 3 routing for control.
+The 'VNets' are deployed locally on each node, after configuration was committed
+from the cluster-wide datacenter SDN administration interface.
+
Main configuration
-------------------
+~~~~~~~~~~~~~~~~~~
+
+The configuration is done at datacenter (cluster-wide) level, it will be saved
+in configuration files located in the shared configuration file system:
+`/etc/pve/sdn`
+
+On the web-interface SDN feature have 3 main sections for the configuration
+
+* SDN: a overview of the SDN state
-The configuration is done at datacenter level.
+* Zones: Create and manage the virtual separated network Zones
-The sdn feature have 4 main sections for the configuration
+* VNets: Create virtual network bridges + subnets management.
-* SDN
+And some options:
-* Zones
+* Controller: For complex setups to control Layer 3 routing
-* Vnets
+* Sub-nets: Used to defined ip networks on VNets.
-* Controller
+* IPAM: Allow to use external tools for IP address management (guest IPs)
+* DNS: Allow to define a DNS server api for registering a virtual guests
+ hostname and IP-addresses
+
+[[pvesdn_config_main_sdn]]
SDN
~~~
-[thumbnail="screenshot/gui-sdn-status.png"]
+This is the main status panel. Here you can see deployment status of zones on
+different nodes.
+
+There is an 'Apply' button, to push and reload local configuration on all
+cluster nodes.
+
+
+[[pvesdn_local_deployment_monitoring]]
+Local Deployment Monitoring
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This is the Main panel, where you can see deployment of zones on differents nodes.
+After applying the configuration through the main SDN web-interface panel,
+the local network configuration is generated locally on each node in
+`/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
-They are an "apply" button, to push && reload local configuration on differents nodes.
+You can monitor the status of local zones and vnets through the main tree.
+[[pvesdn_config_zone]]
Zones
-~~~~~
+-----
-[thumbnail="screenshot/gui-sdn-zone.png"]
+A zone will define a virtually separated network.
-A zone will defined the kind of virtual network you want to defined.
+It can use different technologies for separation:
-it can be
+* VLAN: Virtual LANs are the classic method to sub-divide a LAN
-* vlan
+* QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
-* QinQ (stacked vlan)
+* VXLAN: (layer2 vxlan)
-* vxlan (layer2 vxlan)
+* Simple: Isolated Bridge, simple l3 routing bridge (NAT)
-* bgp-evpn (vxlan with layer3 routing)
+* bgp-evpn: vxlan using layer3 border gateway protocol routing
You can restrict a zone to specific nodes.
-It's also possible to add permissions on a zone, to restrict user
-to use only a specific zone and the vnets in this zone
+It's also possible to add permissions on a zone, to restrict user to use only a
+specific zone and only the VNets in that zone
-Vnets
-~~~~~
+Common options
+~~~~~~~~~~~~~~
-[thumbnail="screenshot/gui-sdn-vnet-evpn.png"]
+The following options are available for all zone types.
-A vnet is a bridge that will be deployed locally on the node,
-for vm communication. (Like a classic vmbrX).
+nodes:: Deploy and allow to use a VNets configured for this Zone only on these
+nodes.
-Vnet properties are:
+ipam:: Optional, if you want to use an ipam tool to manage ips in this zone
-* ID: a 8 characters ID
+dns:: Optional, dns api server.
-* Alias: Optionnal bigger name
+reversedns:: Optional, reverse dns api server.
-* Zone: The associated zone of the vnet
+dnszone:: Optional, dns domain name. Use to register hostname like
+`<hostname>.<domain>`. The dns zone need to be already existing in dns server.
-* Tag: unique vlan or vxlan id
-* ipv4: an anycast ipv4 address (same bridge ip deployed on each node), for bgp-evpn routing only
+[[pvesdn_zone_plugin_simple]]
+Simple Zones
+~~~~~~~~~~~~
-* ipv6: an anycast ipv6 address (same bridge ip deployed on each node), for bgp-evpn routing only
+This is the simplest plugin, it will create an isolated vnet bridge.
+This bridge is not linked to physical interfaces, VM traffic is only
+local to the node(s).
+It can be also used for NAT or routed setup.
+[[pvesdn_zone_plugin_vlan]]
+VLAN Zones
+~~~~~~~~~~
-Controllers
+This plugin will reuse an existing local Linux or OVS bridge,
+and manage VLANs on it.
+The benefit of using SDN module, is that you can create different zones with
+specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
+
+Specific `VLAN` configuration options:
+
+bridge:: Reuse this local bridge or OVS switch, already
+configured on *each* local node.
+
+[[pvesdn_zone_plugin_qinq]]
+QinQ Zones
+~~~~~~~~~~
+
+QinQ is stacked VLAN. The first VLAN tag defined for the zone
+(so called 'service-vlan'), and the second VLAN tag defined for the vnets
+
+NOTE: Your physical network switches must support stacked VLANs!
+
+Specific QinQ configuration options:
+
+bridge:: A local VLAN-aware bridge already configured on each local node
+
+service vlan:: The main VLAN tag of this zone
+
+service vlan protocol:: allow to define a 802.1q (default) or 802.1ad service vlan type.
+
+mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
+For example, you reduce the MTU to `1496` if you physical interface MTU is
+`1500`.
+
+[[pvesdn_zone_plugin_vxlan]]
+VXLAN Zones
~~~~~~~~~~~
-[thumbnail="screenshot/gui-sdn-controller.png"]
+The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
+network (named underlay). It encapsulate layer 2 Ethernet frames within layer
+4 UDP datagrams, using `4789` as the default destination port. You can, for
+example, create a private IPv4 VXLAN network on top of public internet network
+nodes.
+This is a layer2 tunnel only, no routing between different VNets is possible.
+
+Each VNet will have use specific VXLAN id from the range (1 - 16777215).
-Some zone plugins (Currently : bgp-evpn only),
-need an external controller to manage the vnets control-plane.
+Specific EVPN configuration options:
+peers address list:: A list of IPs from all nodes through which you want to
+communicate. Can also be external nodes.
+mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
+lower than the outgoing physical interface.
-Zones Plugins
--------------
-common zone options:
+[[pvesdn_zone_plugin_evpn]]
+EVPN Zones
+~~~~~~~~~~
-* nodes: restrict deploy of the vnets of theses nodes only
+This is the most complex of all supported plugins.
+BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
+have an anycast IP-address and or MAC-address. The bridge IP is the same on each
+node, with this a virtual guest can use that address as gateway.
-Vlan
-~~~~~
+Routing can work across VNets from different zones through a VRF (Virtual
+Routing and Forwarding) interface.
-[thumbnail="screenshot/gui-sdn-zone-vlan.png"]
+Specific EVPN configuration options:
-This is the most simple plugin, it'll reuse an existing local bridge or ovs,
-and manage vlan on it.
-The benefit of using sdn module, is that you can create different zones with specific
-vnets vlan tag, and restrict your customers on their zones.
+VRF VXLAN tag:: This is a vxlan-id used for routing interconnect between vnets,
+it must be different than VXLAN-id of VNets
-specific qinq configuration options:
+controller:: an EVPN-controller need to be defined first (see controller
+plugins section)
-* bridge: a local vlan-aware bridge or ovs switch already configured on each local node
+VNet MAC address:: A unique anycast MAC address for all VNets in this zone.
+ Will be auto-generated if not defined.
-QinQ
-~~~~~
+Exit Nodes:: This is used if you want to define some proxmox nodes, as exit
+ gateway from evpn network through real network. The configured nodes will
+ announce a default route in the EVPN network.
-[thumbnail="screenshot/gui-sdn-zone-qinq.png"]
+Advertise Subnets:: Optional. If you have silent vms/CT (for example, multiples
+ ips by interfaces, and the anycast gateway don't see traffic from theses ips,
+ the ips addresses won't be able to be reach inside the evpn network). This
+ option will announce the full subnet in the evpn network in this case.
-QinQ is stacked vlan.
-you have the first vlan tag defined on the zone (service-vlan), and
-the second vlan tag defined on the vnets
+Exit Nodes local routing:: Optional. This is a special option if you need to
+ reach a vm/ct service from an exit node. (By default, the exit nodes only
+ allow forwarding traffic between real network and evpn network).
-Your physical network switchs need to support stacked vlans !
+MTU:: because VXLAN encapsulation use 50 bytes, the MTU needs to be 50 bytes
+ lower than the maximal MTU of the outgoing physical interface.
-specific qinq configuration options:
-* bridge: a local vlan-aware bridge already configured on each local node
-* service vlan: The main vlan tag of this zone
-* mtu: you need 4 more bytes for the double tag vlan.
- You can reduce the mtu to 1496 if you physical interface mtu is 1500.
+[[pvesdn_config_vnet]]
+VNets
+-----
-Vxlan
-~~~~~
+A `VNet` is in its basic form just a Linux bridge that will be deployed locally
+on the node and used for Virtual Machine communication.
-[thumbnail="screenshot/gui-sdn-zone-vxlan.png"]
+VNet properties are:
-The vxlan plugin will established vxlan tunnel (overlay) on top of an existing network (underlay).
-you can for example, create a private ipv4 vxlan network on top of public internet network nodes.
-This is a layer2 tunnel only, no routing between different vnets is possible.
+ID:: a 8 characters ID to name and identify a VNet
-Each vnet will have a specific vxlan id ( 1 - 16777215 )
+Alias:: Optional longer name, if the ID isn't enough
+Zone:: The associated zone for this VNet
-Specific evpn configuration options:
+Tag:: The unique VLAN or VXLAN id
-* peers address list: an ip list of all nodes where you want to communicate (could be also external nodes)
+VLAN Aware:: Allow to add an extra VLAN tag in the virtual machine or
+ container vNIC configurations or allow the guest OS to manage the VLAN's tag.
-* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
- than the outgoing physical interface.
+[[pvesdn_config_subnet]]
-evpn
-~~~~
+Sub-Nets
+~~~~~~~~
-[thumbnail="screenshot/gui-sdn-zone-evpn.png"]
+A sub-network (subnet or sub-net) allows you to define a specific IP network
+(IPv4 or IPv6). For each VNET, you can define one or more subnets.
-This is the most complex plugin.
+A subnet can be used to:
-BGP-evpn allow to create routable layer3 network.
-The vnet of evpn can have an anycast ip address/mac address.
-The bridge ip is the same on each node, then vm can use
-as gateway.
-The routing is working only across vnets of a specific zone through a vrf.
+* restrict IP-addresses you can define on a specific VNET
+* assign routes/gateway on a VNET in layer 3 zones
+* enable SNAT on a VNET in layer 3 zones
+* auto assign IPs on virtual guests (VM or CT) through IPAM plugin
+* DNS registration through DNS plugins
-Specific evpn configuration options:
+If an IPAM server is associated to the subnet zone, the subnet prefix will be
+automatically registered in the IPAM.
-* vrf vxlan tag: This is a vxlan-id used for routing interconnect between vnets,
- it must be different than vxlan-id of vnets
-* controller: an evpn need to be defined first (see controller plugins section)
+Subnet properties are:
-* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
- than the outgoing physical interface.
+ID:: a cidr network address. Ex: 10.0.0.0/8
+Gateway:: ip address for the default gateway of the network.
+ On layer3 zones (simple/evpn plugins), it'll be deployed on the vnet.
-Controllers Plugins
--------------------
+Snat:: Optional, Enable Snat for layer3 zones (simple/evpn plugins) for this subnet.
+ The subnet source ip will be natted to server outgoing interface/ip.
+ On evpn zone, it's done only on evpn gateway-nodes.
+
+Dnszoneprefix:: Optional, add a prefix to domain registration, like <hostname>.prefix.<domain>
+
+
+[[pvesdn_config_controllers]]
+Controllers
+-----------
-evpn
-~~~~
+Some zone types need an external controller to manage the VNet control-plane.
+Currently this is only required for the `bgp-evpn` zone plugin.
-[thumbnail="screenshot/gui-sdn-controller-evpn.png"]
+[[pvesdn_controller_plugin_evpn]]
+EVPN Controller
+~~~~~~~~~~~~~~~
-For bgp-evpn, we need a controller to manage the control plane.
-The software controller is "frr" router.
-You need to install it on each node where you want to deploy the evpn zone.
+For `BGP-EVPN`, we need a controller to manage the control plane.
+The currently supported software controller is the "frr" router.
+You may need to install it on each node where you want to deploy EVPN zones.
----
-apt install frr
+apt install frr frr-pythontools
----
-configuration options:
+Configuration options:
-*asn: a unique bgp asn number.
- It's recommended to use private asn number (64512 – 65534, 4200000000 – 4294967294)
+asn:: A unique BGP ASN number. It's highly recommended to use private ASN
+number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
+breaking, or get broken, by global routing by mistake.
-*peers: an ip list of all nodes where you want to communicate (could be also external nodes or route reflectors servers)
+peers:: An ip list of all nodes where you want to communicate for the EVPN (could be also
+external nodes or route reflectors servers)
-If you want to route traffic from the sdn bgp-evpn network to external world:
-* gateway-nodes: The proxmox nodes from where the bgp-evpn traffic will exit to external through the nodes default gateway
+[[pvesdn_controller_plugin_BGP]]
+BGP Controller
+~~~~~~~~~~~~~~~
-If you want that gateway nodes don't use the default gateway, but for example, sent traffic to external bgp routers
+The bgp controller is not used directly by a zone.
+You can used it to configure frr to manage bgp peers.
-* gateway-external-peers: 192.168.0.253,192.168.0.254
+For BGP-evpn, it can be use to define a different ASN by node, so doing EBGP.
+Configuration options:
-Local deployment Monitoring
----------------------------
+node:: The node of this BGP controller
-[thumbnail="screenshot/gui-sdn-local-status.png"]
+asn:: A unique BGP ASN number. It's highly recommended to use private ASN
+ number from the range (64512 - 65534) or (4200000000 - 4294967294), as else
+ you could end up breaking, or get broken, by global routing by mistake.
-After apply configuration on the main sdn section,
-the local configuration is generated locally on each node,
-in /etc/network/interfaces.d/sdn, and reloaded.
+peers:: An IP list of peers you want to communicate with for the underlying
+ BGP network.
-You can monitor the status of local zones && vnets through the main tree.
+ebgp:: If your peer's remote-AS is different, it's enabling EBGP.
+loopback:: If you want to use a loopback or dummy interface as source for the
+ evpn network. (for multipath)
+ebgp-mutltihop:: if the peers are not directly connected or use loopback, you can increase the
+ number of hops to reach them.
+
+[[pvesdn_config_ipam]]
+IPAMs
+-----
+IPAM (IP address management) tools, are used to manage/assign ips on your devices on the network.
+It can be used to find free ip address when you create a vm/ct for example (not yet implemented).
+
+An IPAM is associated to 1 or multiple zones, to provide ip addresses for all subnets defined in this zone.
+
+
+[[pvesdn_ipam_plugin_pveipam]]
+{pve} IPAM plugin
+~~~~~~~~~~~~~~~~~
+
+This is the default internal IPAM for your proxmox cluster if you don't have
+external ipam software
+
+[[pvesdn_ipam_plugin_phpipam]]
+phpIPAM plugin
+~~~~~~~~~~~~~~
+https://phpipam.net/
+
+You need to create an application in phpipam, and add an api token with admin
+permission
+
+phpIPAM properties are:
+
+url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
+token:: An API access token
+section:: An integer ID. Sections are group of subnets in phpIPAM. Default
+ installations use `sectionid=1` for customers.
+
+[[pvesdn_ipam_plugin_netbox]]
+Netbox IPAM plugin
+~~~~~~~~~~~~~~~~~~
+
+NetBox is an IP address management (IPAM) and data center infrastructure
+management (DCIM) tool, see the source code repository for details:
+https://github.com/netbox-community/netbox
+
+You need to create an api token in netbox
+https://netbox.readthedocs.io/en/stable/api/authentication
+
+NetBox properties are:
+
+url:: The REST API endpoint: `http://yournetbox.domain.com/api`
+token:: An API access token
+
+[[pvesdn_config_dns]]
+DNS
+---
+
+The DNS plugin in {pve} SDN is used to define a DNS API server for registration
+of your hostname and IP-address. A DNS configuration is associated with one or
+more zones, to provide DNS registration for all the sub-net IPs configured for
+a zone.
+
+[[pvesdn_dns_plugin_powerdns]]
+PowerDNS plugin
+~~~~~~~~~~~~~~~
+https://doc.powerdns.com/authoritative/http-api/index.html
+
+You need to enable the webserver and the API in your PowerDNS config:
+
+----
+api=yes
+api-key=arandomgeneratedstring
+webserver=yes
+webserver-port=8081
+----
+
+Powerdns properties are:
+
+url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
+key:: An API access key
+ttl:: The default TTL for records
+
+
+Examples
+--------
+
+[[pvesdn_setup_example_vlan]]
+VLAN Setup Example
+~~~~~~~~~~~~~~~~~~
+
+TIP: While we show plain configuration content here, almost everything should
+be configurable using the web-interface only.
+
+Node1: /etc/network/interfaces
-Vlan setup example
-------------------
-node1: /etc/network/interfaces
----
auto vmbr0
iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
address 192.168.0.1/24
source /etc/network/interfaces.d/*
-
----
-node2: /etc/network/interfaces
+Node2: /etc/network/interfaces
----
auto vmbr0
iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
source /etc/network/interfaces.d/*
----
-create an vlan zone
+Create a VLAN zone named `myvlanzone':
----
-id: mylanzone
+id: myvlanzone
bridge: vmbr0
----
-create a vnet1 with vlan-id 10
+Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
+`myvlanzone' as it's zone.
----
id: myvnet1
tag: 10
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+Apply the configuration through the main SDN panel, to create VNets locally on
+each nodes.
+Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
-create a vm1, with 1 nic on vnet1 on node1
+Use the following network configuration for this VM:
----
auto eth0
iface eth0 inet static
- address 10.0.3.100/24
+ address 10.0.3.100/24
----
-create a vm2, with 1 nic on vnet1 on node2
+Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
+`myvnet1' as vm1.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
- address 10.0.3.101/24
+ address 10.0.3.101/24
----
-Then, you should be able to ping between between vm1 && vm2
+Then, you should be able to ping between both VMs over that network.
-QinQ setup example
-------------------
-node1: /etc/network/interfaces
+[[pvesdn_setup_example_qinq]]
+QinQ Setup Example
+~~~~~~~~~~~~~~~~~~
+
+TIP: While we show plain configuration content here, almost everything should
+be configurable using the web-interface only.
+
+Node1: /etc/network/interfaces
+
----
auto vmbr0
iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
source /etc/network/interfaces.d/*
----
-node2: /etc/network/interfaces
+Node2: /etc/network/interfaces
----
auto vmbr0
iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
source /etc/network/interfaces.d/*
----
-create an qinq zone1 with service vlan 20
+Create an QinQ zone named `qinqzone1' with service VLAN 20
----
id: qinqzone1
service vlan: 20
----
-create an qinq zone2 with service vlan 30
+Create another QinQ zone named `qinqzone2' with service VLAN 30
----
id: qinqzone2
service vlan: 30
----
-create a vnet1 with customer vlan-id 100 on qinqzone1
+Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
+created `qinqzone1' zone.
----
id: myvnet1
tag: 100
----
-create a vnet2 with customer vlan-id 100 on qinqzone2
+Create a `myvnet2' with customer VLAN-id 100 on the previously created
+`qinqzone2' zone.
----
id: myvnet2
-zone: qinqzone1
+zone: qinqzone2
tag: 100
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+Apply the configuration on the main SDN web-interface panel to create VNets
+locally on each nodes.
+Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
-create a vm1, with 1 nic on vnet1 on node1
+Use the following network configuration for this VM:
----
auto eth0
address 10.0.3.100/24
----
-create a vm2, with 1 nic on vnet1 on node2
+Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
+`myvnet1' as vm1.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
address 10.0.3.101/24
----
-create a vm3, with 1 nic on vnet2 on node1
+Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
+`myvnet2'.
+
+Use the following network configuration for this VM:
----
auto eth0
address 10.0.3.102/24
----
-create a vm4, with 1 nic on vnet2 on node2
+Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
+`myvnet2' as vm3.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
address 10.0.3.103/24
----
-Then, you should be able to ping between between vm1 && vm2
-vm3 && vm4 could ping together
+Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
+between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
+or 'vm4', as they are on a different zone with different service-vlan.
+
-but vm1 && vm2 couldn't ping vm3 && vm4,
-as it's a different zone, with different service vlan
+[[pvesdn_setup_example_vxlan]]
+VXLAN Setup Example
+~~~~~~~~~~~~~~~~~~~
+TIP: While we show plain configuration content here, almost everything should
+be configurable using the web-interface only.
-Vxlan setup example
--------------------
node1: /etc/network/interfaces
+
----
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1/24
gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
mtu 1500
source /etc/network/interfaces.d/*
iface vmbr0 inet static
address 192.168.0.2/24
gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
mtu 1500
source /etc/network/interfaces.d/*
iface vmbr0 inet static
address 192.168.0.3/24
gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
+ bridge-ports eno1
+ bridge-stp off
+ bridge-fd 0
mtu 1500
source /etc/network/interfaces.d/*
----
-create an vxlan zone
+Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
+50 bytes of the VXLAN header can fit. Add all previously configured IPs from
+the nodes as peer address list.
----
id: myvxlanzone
mtu: 1450
----
-create first vnet
+Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
+previously.
----
id: myvnet1
tag: 100000
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+Apply the configuration on the main SDN web-interface panel to create VNets
+locally on each nodes.
+Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
-create a vm1, with 1 nic on vnet1 on node2
+Use the following network configuration for this VM, note the lower MTU here.
----
auto eth0
mtu 1450
----
-create a vm2, with 1 nic on vnet1 on node3
+Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
+`myvnet1' as vm1.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
mtu 1450
----
-Then, you should be able to ping between between vm1 && vm2
+Then, you should be able to ping between between 'vm1' and 'vm2'.
+[[pvesdn_setup_example_evpn]]
+EVPN Setup Example
+~~~~~~~~~~~~~~~~~~
-EVPN setup example
-------------------
node1: /etc/network/interfaces
----
source /etc/network/interfaces.d/*
----
-create a evpn controller
+Create a EVPN controller, using a private ASN number and above node addreesses
+as peers.
----
id: myevpnctl
asn: 65000
peers: 192.168.0.1,192.168.0.2,192.168.0.3
-gateway nodes: node1,node2
----
-create an evpn zone
+Create an EVPN zone named `myevpnzone' using the previously created
+EVPN-controller Define 'node1' and 'node2' as exit nodes.
----
id: myevpnzone
vrf vxlan tag: 10000
controller: myevpnctl
mtu: 1450
+vnet mac address: 32:F4:05:FE:6C:0A
+exitnodes: node1,node2
----
-create first vnet
-
+Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
----
id: myvnet1
zone: myevpnzone
tag: 11000
-ipv4: 10.0.1.1/24
-mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
----
-create second vnet
+Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on vnet1
+
+----
+subnet: 10.0.1.0/24
+gateway: 10.0.1.1
+----
+
+Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
+different IPv4 CIDR network.
----
id: myvnet2
zone: myevpnzone
tag: 12000
-ipv4: 10.0.2.1/24
-mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet
----
-Apply the configuration on the main sdn section, to create vnets locally on each nodes,
-and generate frr config.
+Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
+----
+subnet: 10.0.2.0/24
+gateway: 10.0.2.1
+----
-create a vm1, with 1 nic on vnet1 on node2
+Apply the configuration on the main SDN web-interface panel to create VNets
+locally on each nodes and generate the FRR config.
+
+Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+
+Use the following network configuration for this VM:
----
auto eth0
mtu 1450
----
-create a vm2, with 1 nic on vnet2 on node3
+Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
+`myvnet2'.
+
+Use the following network configuration for this VM:
+
----
auto eth0
iface eth0 inet static
Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
-from vm2 on node3, if you ping an external ip, the packet will go
-to the vnet2 gateway, then will be routed to gateway nodes (node1 or node2)
-then the packet will be routed to the node1 or node2 default gw.
+If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
+will go to the configured 'myvnet2' gateway, then will be routed to the exit
+nodes ('node1' or 'node2') and from there it will leave those nodes over the
+default gateway configured on node1 or node2.
+
+NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
+'10.0.2.0/24' network to node1, node2 on your external gateway, so that the
+public network can reply back.
+
+If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
+and 10.0.2.0/24 in this example), will be announced dynamically.
+
+
+Notes
+-----
-Of course you need to add reverse routes to 10.0.1.0/24 && 10.0.2.0/24 to node1,node2 on your external gateway.
+VXLAN IPSEC Encryption
+~~~~~~~~~~~~~~~~~~~~~~
+If you need to add encryption on top of VXLAN, it's possible to do so with
+IPSEC through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
+or 80 bytes (IPv6) to handle encryption.
-If you have configured an external bgp router, the bgp-evpn routes (10.0.1.0/24 && 10.0.2.0/24),
-will be announced dynamically.
+So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
++ 50 (VXLAN) == 1500).
+
+.Install strongswan
+----
+apt install strongswan
+----
+
+Add configuration in `/etc/ipsec.conf'. We only need to encrypt traffic from
+the VXLAN UDP port '4789'.
+
+----
+conn %default
+ ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
+ esp=aes256-sha1!
+ leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
+
+conn output
+ rightsubnet=%dynamic[udp/4789]
+ right=%any
+ type=transport
+ authby=psk
+ auto=route
+
+conn input
+ leftsubnet=%dynamic[udp/4789]
+ type=transport
+ authby=psk
+ auto=route
+----
+
+Then generate a preshared key with
+
+----
+openssl rand -base64 128
+----
+
+and copy the key in `/etc/ipsec.secrets' so that the file content looks like:
+
+----
+: PSK <generatedbase64key>
+----
+You need to copy the PSK and the config on other nodes.
projects / pve-docs.git / blobdiff