[[chapter_pvesdn]]
-Software Defined Network
+Software-Defined Network
========================
ifndef::manvolnum[]
:pve-toplevel:
endif::manvolnum[]
-The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
-virtual networks (vnets) at datacenter level.
+The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
+virtual networks (VNets) at the datacenter level.
WARNING: SDN is currently an **experimental feature** in {pve}. This
-Documentation for it is also still under development, ask on our
+documentation for it is also still under development. Ask on our
xref:getting_help[mailing lists or in the forum] for questions and feedback.
Installation
------------
-To enable the experimental SDN integration, you need to install
-"libpve-network-perl" package
+To enable the experimental Software-Defined Network (SDN) integration, you need
+to install the `libpve-network-perl` package on every node:
----
+apt update
apt install libpve-network-perl
----
-You need to have `ifupdown2` package installed on each node to manage local
-configuration reloading without reboot:
+NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
+default. If you originally installed your system with an older version, you need
+to explicitly install the `ifupdown2` package.
-----
-apt install ifupdown2
-----
+After installation, you need to add the following line to the end of the
+`/etc/network/interfaces` configuration file, so that the SDN configuration gets
+included and activated.
-You need to add
----
source /etc/network/interfaces.d/*
----
-at the end of /etc/network/interfaces to have the sdn config included
-Basic Overview
---------------
+[[pvesdn_overview]]
+Overview
+--------
-The {pve} SDN allows separation and fine grained control of Virtual Guests
-networks, using flexible software controlled configurations.
+The {pve} SDN allows for separation and fine-grained control of virtual guest
+networks, using flexible, software-controlled configurations.
-Separation consists of zones, a zone is it's own virtual separated network area.
-A 'VNet' is a type of a virtual network connected to a zone. Depending on which
-type or plugin the zone uses it can behave differently and offer different
-features, advantages or disadvantages.
-Normally a 'VNet' shows up as a common Linux bridge with either a VLAN or
-'VXLAN' tag, but some can also use layer 3 routing for control.
-The 'VNets' are deployed locally on each node, after configuration was committed
-from the cluster wide datacenter SDN administration interface.
+Separation is managed through *zones*, virtual networks (*VNets*), and
+*subnets*. A zone is its own virtually separated network area. A VNet is a
+virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
+Depending on the type of the zone, the network behaves differently and offers
+specific features, advantages, and limitations.
-Main configuration
-~~~~~~~~~~~~~~~~~~
+Use cases for SDN range from an isolated private network on each individual node
+to complex overlay networks across multiple PVE clusters on different locations.
-The configuration is done at datacenter (cluster-wide) level, it will be saved
-in configuration files located in the shared configuration file system:
-`/etc/pve/sdn`
+After configuring an VNet in the cluster-wide datacenter SDN administration
+interface, it is available as a common Linux bridge, locally on each node, to be
+assigned to VMs and Containers.
-On the web-interface SDN feature have 3 main sections for the configuration
-* SDN: a overview of the SDN state
+[[pvesdn_main_configuration]]
+Main Configuration
+~~~~~~~~~~~~~~~~~~
-* Zones: Create and manage the virtual separated network Zones
+Configuration is done at the web UI at datacenter level and is saved in files
+located in the shared configuration file system at `/etc/pve/sdn`.
-* VNets: Create virtual network bridges + subnets management.
+On the web interface, SDN features the following sections:
-And some options:
+* SDN:: Here you get an overview of the current active SDN state, and you can
+ apply all pending changes to the whole cluster.
-* Controller: For complex setups to control Layer 3 routing
+* xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
+ network zones
-* Ipams: Allow to use external tools for ip managements (vm/ct ips)
+* xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and
+ manage subnets
-* Dns: Allow to define a dns server api for register vm/ct hostname/ip addresses
+The Options category allows adding and managing additional services to be used
+in your SDN setup.
+* xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
+ in complex setups
-[[pvesdn_config_main_sdn]]
+* xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
+ guests
+
+* xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
+ virtual guests' hostname and IP
+ addresses
+[[pvesdn_config_main_sdn]]
SDN
~~~
-This is the main status panel. Here you can see deployment status of zones on
-different nodes.
-
-There is an 'Apply' button, to push and reload local configuration on all
-cluster nodes nodes.
-
-
-[[pvesdn_local_deployment_monitoring]]
-Local Deployment Monitoring
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-After applying the configuration through the main SDN web-interface panel,
-the local network configuration is generated locally on each node in
-`/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
-
-You can monitor the status of local zones and vnets through the main tree.
+This is the main status panel. Here you can see the deployment status of zones
+on different nodes.
+Pressing the 'Apply' button reloads the local configuration on all cluster
+nodes.
[[pvesdn_config_zone]]
Zones
-----
-A zone will define a virtually separated network.
-
-It can use different technologies for separation:
+A zone defines a virtually separated network. Zones are restricted to
+specific nodes and assigned permissions, in order to restrict users to a certain
+zone and its contained VNets.
-* VLAN: Virtual LANs are the classic method to sub-divide a LAN
+Different technologies can be used for separation:
-* QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
+* Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
-* VXLAN: (layer2 vxlan)
+* VLAN: Virtual LANs are the classic method of subdividing a LAN
-* Simple: Isolated Bridge, simple l3 routing bridge (NAT)
+* QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
-* bgp-evpn: vxlan using layer3 border gateway protocol routing
+* VXLAN: Layer 2 VXLAN network via a UDP tunnel
-You can restrict a zone to specific nodes.
+* EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
-It's also possible to add permissions on a zone, to restrict user to use only a
-specific zone and only the VNets in that zone
-Common options
+[[pvesdn_config_common_options]]
+Common Options
~~~~~~~~~~~~~~
-nodes:: Deploy and allow to use a VNets configured for this Zone only on these
-nodes.
+The following options are available for all zone types:
+
+Nodes:: The nodes which the zone and associated VNets should be deployed on.
-Ipam:: Optional, if you want to use an ipam tool to manage ips in this zone
+IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
+ zone. Optional, defaults to `pve`.
-Dns:: Optional, dns api server.
+DNS:: DNS API server. Optional.
-ReverseDns:: Optional, reverse dns api server.
+ReverseDNS:: Reverse DNS API server. Optional.
-Dnszone:: Optional, dns domain name. Use to register hostname like <hostname>.<domain>
- The dns zone need to be already existing in dns server.
+DNSZone:: DNS domain name. Used to register hostnames, such as
+ `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
[[pvesdn_zone_plugin_simple]]
Simple Zones
~~~~~~~~~~~~
-This is the simplest plugin, it will create an isolated vnet bridge.
-This bridge is not linked to physical interfaces, VM traffic is only
-local to the node(s).
-It can be also used for NAT or routed setup.
+This is the simplest plugin. It will create an isolated VNet bridge. This
+bridge is not linked to a physical interface, and VM traffic is only local on
+each the node.
+It can be used in NAT or routed setups.
+
[[pvesdn_zone_plugin_vlan]]
VLAN Zones
~~~~~~~~~~
-This plugin will reuse an existing local Linux or OVS bridge,
-and manage VLANs on it.
-The benefit of using SDN module, is that you can create different zones with
-specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
+The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
+node's physical interface. It uses VLAN tagging defined in the VNet to isolate
+the network segments. This allows connectivity of VMs between different nodes.
+
+VLAN zone configuration options:
-Specific `VLAN` configuration options:
+Bridge:: The local bridge or OVS switch, already configured on *each* node that
+ allows node-to-node connection.
-bridge:: Reuse this local bridge or OVS switch, already
-configured on *each* local node.
[[pvesdn_zone_plugin_qinq]]
QinQ Zones
~~~~~~~~~~
-QinQ is stacked VLAN. The first VLAN tag defined for the zone
-(so called 'service-vlan'), and the second VLAN tag defined for the vnets
+QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
+isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
+whereas the inner VLAN tag is defined by the VNet.
+
+NOTE: Your physical network switches must support stacked VLANs for this
+configuration.
-NOTE: Your physical network switches must support stacked VLANs!
+QinQ zone configuration options:
-Specific QinQ configuration options:
+Bridge:: A local, VLAN-aware bridge that is already configured on each local
+ node
-bridge:: A local VLAN-aware bridge already configured on each local node
+Service VLAN:: The main VLAN tag of this zone
-service vlan:: The main VLAN tag of this zone
+Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
+ 802.1ad service VLAN type.
+
+MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
+ For example, you must reduce the MTU to `1496` if you physical interface MTU is
+ `1500`.
-mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
-For example, you reduce the MTU to `1496` if you physical interface MTU is
-`1500`.
[[pvesdn_zone_plugin_vxlan]]
VXLAN Zones
~~~~~~~~~~~
-The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
-network (named underlay). It encapsulate layer 2 Ethernet frames within layer
-4 UDP datagrams, using `4789` as the default destination port. You can, for
-example, create a private IPv4 VXLAN network on top of public internet network
-nodes.
-This is a layer2 tunnel only, no routing between different VNets is possible.
+The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
+(underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
+datagrams using the default destination port `4789`.
+
+You have to configure the underlay network yourself to enable UDP connectivity
+between all peers.
-Each VNet will have use specific VXLAN id from the range (1 - 16777215).
+You can, for example, create a VXLAN overlay network on top of public internet,
+appearing to the VMs as if they share the same local Layer 2 network.
-Specific EVPN configuration options:
+WARNING: VXLAN on its own does does not provide any encryption. When joining
+ multiple sites via VXLAN, make sure to establish a secure connection between
+ the site, for example by using a site-to-site VPN.
-peers address list:: A list of IPs from all nodes through which you want to
-communicate. Can also be external nodes.
+VXLAN zone configuration options:
+
+Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
+ can be external nodes reachable at this IP address.
+ All nodes in the cluster need to be mentioned here.
+
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ lower than the outgoing physical interface.
-mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
-lower than the outgoing physical interface.
[[pvesdn_zone_plugin_evpn]]
EVPN Zones
~~~~~~~~~~
-This is the most complex of all supported plugins.
+The EVPN zone creates a routable Layer 3 network, capable of spanning across
+multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
+the routing protocol.
-BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
-have an anycast IP-address and or MAC-address. The bridge IP is the same on each
-node, with this a virtual guest can use that address as gateway.
+The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
+IP is the same on each node, meaning a virtual guest can use this address as
+gateway.
Routing can work across VNets from different zones through a VRF (Virtual
Routing and Forwarding) interface.
-Specific EVPN configuration options:
+EVPN zone configuration options:
+
+VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
+ It must be different than the VXLAN-ID of the VNets.
-VRF VXLAN Tag:: This is a vxlan-id used for routing interconnect between vnets,
-it must be different than VXLAN-id of VNets
+Controller:: The EVPN-controller to use for this zone. (See controller plugins
+ section).
-controller:: an EVPN-controller need to be defined first (see controller
-plugins section)
+VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
+ zone. Will be auto-generated if not defined.
+Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
+ network, through the real network. The configured nodes will announce a
+ default route in the EVPN network. Optional.
-Exit Nodes:: This is used if you want to defined some proxmox nodes, as
- exit gateway from evpn network through real network. This nodes
- will announce a default route in the evpn network.
+Primary Exit Node:: If you use multiple exit nodes, force traffic through this
+ primary exit node, instead of load-balancing on all nodes. Optional but
+ necessary if you want to use SNAT or if your upstream router doesn't support
+ ECMP.
-mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
-lower than the outgoing physical interface.
+Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
+ service from an exit node. (By default, the exit nodes only allow forwarding
+ traffic between real network and EVPN network). Optional.
+Advertise Subnets:: Announce the full subnet in the EVPN network.
+ If you have silent VMs/CTs (for example, if you have multiple IPs and the
+ anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
+ able to be reached inside the EVPN network). Optional.
-[[pvesdn_config_vnet]]
+Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
+ packets. This is required if you use floating IPs in your VMs (IP and MAC
+ addresses are being moved between systems). Optional.
+
+Route-target Import:: Allows you to import a list of external EVPN route
+ targets. Used for cross-DC or different EVPN network interconnects. Optional.
+
+MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
+ less than the maximal MTU of the outgoing physical interface. Optional,
+ defaults to 1450.
+
+
+[[pvesdn_config_vnets]]
VNets
-----
-A `VNet` is in its basic form just a Linux bridge that will be deployed locally
-on the node and used for Virtual Machine communication.
+After creating a virtual network (VNet) through the SDN GUI, a local network
+interface with the same name is available on each node. To connect a guest to the
+VNet, assign the interface to the guest and set the IP address accordingly.
-VNet properties are:
+Depending on the zone, these options have different meanings and are explained
+in the respective zone section in this document.
-ID:: a 8 characters ID to name and identify a VNet
+WARNING: In the current state, some options may have no effect or won't work in
+certain zones.
-Alias:: Optional longer name, if the ID isn't enough
+VNet configuration options:
+
+ID:: An up to 8 character ID to identify a VNet
+
+Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
Zone:: The associated zone for this VNet
-Tag:: The unique VLAN or VXLAN id
+Tag:: The unique VLAN or VXLAN ID
-VLAN Aware:: Allow to add an extra VLAN tag in the virtual machine or
- container vNIC configurations or allow the guest OS to manage the VLAN's tag.
+VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
+ in the quest.
-[[pvesdn_config_subnet]]
+[[pvesdn_config_subnet]]
Subnets
-~~~~~~~
-
-For each Vnet, you can define 1 or multiple subnets to define an ip network (ipv4 or ipv6).
+-------
-It can be used to restrict ip addresses you can define on a specific vnet,
-assign routes/gateway on vnet in layer3 zones,
-enable snat in layer 3 zones,
-auto assign ips on vm/ct through ipam plugin && dns registration through dns plugins.
+A subnet define a specific IP range, described by the CIDR network address.
+Each VNet, can have one or more subnets.
-If an ipam server is associated to the subnet zone, the subnet prefix will be automatically
-registered in the ipam.
+A subnet can be used to:
+* Restrict the IP addresses you can define on a specific VNet
+* Assign routes/gateways on a VNet in layer 3 zones
+* Enable SNAT on a VNet in layer 3 zones
+* Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
+* DNS registration through DNS plugins
-Subnet properties are:
+If an IPAM server is associated with the subnet zone, the subnet prefix will be
+automatically registered in the IPAM.
-ID:: a cidr network address. Ex: 10.0.0.0/8
+Subnet configuration options:
-Gateway:: ip address for the default gateway of the network.
- On layer3 zones (simple/evpn plugins), it'll be deployed on the vnet.
-
-Snat:: Optional, Enable Snat for layer3 zones (simple/evpn plugins) for this subnet.
- The subnet source ip will be natted to server outgoing interface/ip.
- On evpn zone, it's done only on evpn gateway-nodes.
+ID:: A CIDR network address, for example 10.0.0.0/8
-Dnszoneprefix:: Optional, add a prefix to domain registration, like <hostname>.prefix.<domain>
+Gateway:: The IP address of the network's default gateway. On layer 3 zones
+ (Simple/EVPN plugins), it will be deployed on the VNet.
+SNAT:: Enable Source NAT which allows VMs from inside a
+ VNet to connect to the outside network by forwarding the packets to the nodes
+ outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
+ Optional.
+DNS Zone Prefix:: Add a prefix to the domain registration, like
+ <hostname>.prefix.<domain> Optional.
[[pvesdn_config_controllers]]
Controllers
-----------
-Some zone types need an external controller to manage the VNet control-plane.
-Currently this is only required for the `bgp-evpn` zone plugin.
+Some zones implement a separated control and data plane that require an external
+external controller to manage the VNet's control plane.
+
+Currently, only the `EVPN` zone requires an external controller.
+
[[pvesdn_controller_plugin_evpn]]
EVPN Controller
~~~~~~~~~~~~~~~
-For `BGP-EVPN`, we need a controller to manage the control plane.
-The currently supported software controller is the "frr" router.
-You may need to install it on each node where you want to deploy EVPN zones.
+The `EVPN`, zone requires an external controller to manage the control plane.
+The EVPN controller plugin configures the Free Range Routing (frr) router.
+
+To enable the EVPN controller, you need to install frr on every node that shall
+participate in the EVPN zone.
----
apt install frr frr-pythontools
----
-Configuration options:
+EVPN controller configuration options:
-asn:: A unique BGP ASN number. It's highly recommended to use private ASN
-number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
-breaking, or get broken, by global routing by mistake.
+ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
+ breaking global routing by mistake.
-peers:: An ip list of all nodes where you want to communicate for the EVPN (could be also
-external nodes or route reflectors servers)
+Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
+ external nodes or route reflector servers)
[[pvesdn_controller_plugin_BGP]]
BGP Controller
-~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~
+
+The BGP controller is not used directly by a zone.
+You can use it to configure FRR to manage BGP peers.
+
+For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
+It can also be used to export EVPN routes to an external BGP peer.
+
+NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
+controller.
+
+BGP controller configuration options:
+
+Node:: The node of this BGP controller
-The bgp controller is not used directly by a zone.
-You can used it to configure frr to manage bgp peers.
+ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
+ number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
+ you could break global routing by mistake.
-For Bgp-evpn, it can be use to define a different ASN by node,
-so doing ebgp.
+Peer:: A list of peer IP addresses you want to communicate with using the
+ underlying BGP network.
-Configuration options:
+EBGP:: If your peer's remote-AS is different, this enables EBGP.
+
+Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
+
+ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
+ not directly connected or they use loopback.
+
+bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
+
+
+[[pvesdn_controller_plugin_ISIS]]
+ISIS Controller
+~~~~~~~~~~~~~~~
-asn:: A unique BGP ASN number. It's highly recommended to use private ASN
-number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
-breaking, or get broken, by global routing by mistake.
+The ISIS controller is not used directly by a zone.
+You can use it to configure FRR to export EVPN routes to an ISIS domain.
-peers:: An ip list of peers where you want to communicate for the underlay
-BGP network
+ISIS controller configuration options:
-ebgp:: if your peers remote-as is different, it's enabling ebgp.
+Node:: The node of this ISIS controller.
-node:: the node of this bgp controller
+Domain:: A unique ISIS domain.
-loopback:: If you want to use a loopback or dummy interface as source
- for the evpn network. (for multipath)
+Network Entity Title:: A Unique ISIS network address that identifies this node.
+
+Interfaces:: A list of physical interface(s) used by ISIS.
+
+Loopback:: Use a loopback or dummy interface as the source of the EVPN network
+ (for multipath).
[[pvesdn_config_ipam]]
-Ipams
------
-IPAM (IP address management) tools, are used to manage/assign ips on your devices on the network.
-It can be used to find free ip address when you create a vm/ct for example (not yet implemented).
+IPAM
+----
-An IPAM is associated to 1 or multiple zones, to provide ip addresses for all subnets defined in this zone.
+IP Address Management (IPAM) tools manage the IP addresses of clients on the
+network. SDN in {pve} uses IPAM for example to find free IP addresses for new
+guests.
+
+A single IPAM instance can be associated with one or more zones.
[[pvesdn_ipam_plugin_pveipam]]
-PVEIpam plugin
-~~~~~~~~~~~~~~
+PVE IPAM Plugin
+~~~~~~~~~~~~~~~
+
+The default built-in IPAM for your {pve} cluster.
+
+
+[[pvesdn_ipam_plugin_netbox]]
+NetBox IPAM Plugin
+~~~~~~~~~~~~~
+
+link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
+Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
+
+To integrate NetBox with {pve} SDN, create an API token in NetBox as described
+here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
+
+The NetBox configuration properties are:
+
+URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
+
+Token:: An API access token
-This is the default internal ipam for your proxmox cluster if you don't have external ipam software
[[pvesdn_ipam_plugin_phpipam]]
-PHPIpam plugin
+phpIPAM Plugin
~~~~~~~~~~~~~~
-https://phpipam.net/
-You need to create an application in phpipam, and add an api token with admin permission
+In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
+an API token with admin privileges to the application.
-PHPipam properties are:
+The phpIPAM configuration properties are:
-* Url: The rest api url : http://phpipam.domain.com/api/<appname>/
-* Token: your api token
-* Section: An integer id. Sections are group of subnets in phpipam.
- Default install have sectionid=1 for customers
-
-[[pvesdn_ipam_plugin_netbox]]
-Netbox Ipam plugin
-~~~~~~~~~~~~~~~~~~
-https://github.com/netbox-community/netbox
+URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
-you need to create an api token in netbox
-https://netbox.readthedocs.io/en/stable/api/authentication
+Token:: An API access token
-PHPipam properties are:
+Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
+ installations use `sectionid=1` for customers.
-Url:: The rest api url: http://yournetbox.domain.com/api
-Token:: your api token
[[pvesdn_config_dns]]
-Dns
+DNS
---
-Dns is used to define a dns api server for registration of your hostname/ip address
-an DNS is associated to 1 or multiple zones, to provide dns registration
-for all ips in subnets defined in this zone.
+
+The DNS plugin in {pve} SDN is used to define a DNS API server for registration
+of your hostname and IP address. A DNS configuration is associated with one or
+more zones, to provide DNS registration for all the subnet IPs configured for
+a zone.
[[pvesdn_dns_plugin_powerdns]]
-Powerdns plugin
+PowerDNS Plugin
~~~~~~~~~~~~~~~
https://doc.powerdns.com/authoritative/http-api/index.html
-you need to enable webserver && api in your powerdns config:
+You need to enable the web server and the API in your PowerDNS config:
----
api=yes
webserver-port=8081
----
-Powerdns properties are:
+The PowerDNS configuration options are:
+
+url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
+
+key:: An API access key
-Url:: The rest api url: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
-key:: the api key
-ttl:: default ttl for records
+ttl:: The default TTL for records
+[[pvesdn_setup_examples]]
Examples
--------
-[[pvesdn_setup_example_vlan]]
-VLAN Setup Example
-~~~~~~~~~~~~~~~~~~
+This section presents multiple configuration examples tailored for common SDN
+use cases. It aims to offer tangible implementations, providing additional
+details to enhance comprehension of the available configuration options.
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
-Node1: /etc/network/interfaces
+[[pvesdn_setup_example_simple]]
+Simple Zone Example
+~~~~~~~~~~~~~~~~~~~
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+Simple zone networks create an isolated network for quests on a single host to
+connect to each other.
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.1/24
+TIP: connection between quests are possible if all quests reside on a same host
+but cannot be reached on other nodes.
-source /etc/network/interfaces.d/*
-----
+* Create a simple zone named `simple`.
+* Add a VNet names `vnet1`.
+* Create a Subnet with a gateway and the SNAT option enabled.
+* This creates a network bridge `vnet1` on the node. Assign this bridge to the
+ quests that shall join the network and configure an IP address.
-Node2: /etc/network/interfaces
+The network interface configuration in two VMs may look like this which allows
+them to communicate via the 10.0.1.0/24 network.
----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+allow-hotplug ens19
+iface ens19 inet static
+ address 10.0.1.14/24
+----
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.2/24
+----
+allow-hotplug ens19
+iface ens19 inet static
+ address 10.0.1.15/24
+----
+
+
+[[pvesdn_setup_example_nat]]
+Source NAT Example
+~~~~~~~~~~~~~~~~~~
+
+If you want to allow outgoing connections for quests in the simple network zone
+the simple zone offers a Source NAT (SNAT) option.
+
+Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
+Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
-source /etc/network/interfaces.d/*
----
+Subnet: 172.16.0.0/24
+Gateway: 172.16.0.1
+SNAT: checked
+----
+
+In the quests configure the static IP address inside the subnet's IP range.
+
+The node itself will join this network with the Gateway IP '172.16.0.1' and
+function as the NAT gateway for quests within the subnet range.
+
+
+[[pvesdn_setup_example_vlan]]
+VLAN Setup Example
+~~~~~~~~~~~~~~~~~~
+
+When VMs on different nodes need to communicate through an isolated network, the
+VLAN zone allows network level isolation using VLAN tags.
-Create a VLAN zone named `myvlanzone':
+Create a VLAN zone named `myvlanzone`:
----
-id: myvlanzone
-bridge: vmbr0
+ID: myvlanzone
+Bridge: vmbr0
----
-Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
-`myvlanzone' as it's zone.
+Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
+`myvlanzone`.
----
-id: myvnet1
-zone: myvlanzone
-tag: 10
+ID: myvnet1
+Zone: myvlanzone
+Tag: 10
----
Apply the configuration through the main SDN panel, to create VNets locally on
-each nodes.
+each node.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
Use the following network configuration for this VM:
address 10.0.3.100/24
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
-`myvnet1' as vm1.
+Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
+`myvnet1` as vm1.
Use the following network configuration for this VM:
address 10.0.3.101/24
----
-Then, you should be able to ping between both VMs over that network.
+Following this, you should be able to ping between both VMs using that network.
[[pvesdn_setup_example_qinq]]
QinQ Setup Example
~~~~~~~~~~~~~~~~~~
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
-Node1: /etc/network/interfaces
+This example configures two QinQ zones and adds two VMs to each zone to
+demonstrate the additional layer of VLAN tags which allows the configuration of
+more isolated VLANs.
-----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
+A typical use case for this configuration is a hosting provider that provides an
+isolated network to customers for VM communication but isolates the VMs from
+other customers.
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.1/24
+Create a QinQ zone named `qinqzone1` with service VLAN 20
-source /etc/network/interfaces.d/*
----
-
-Node2: /etc/network/interfaces
-
+ID: qinqzone1
+Bridge: vmbr0
+Service VLAN: 20
----
-auto vmbr0
-iface vmbr0 inet manual
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- bridge-vlan-aware yes
- bridge-vids 2-4094
-
-#management ip on vlan100
-auto vmbr0.100
-iface vmbr0.100 inet static
- address 192.168.0.2/24
-source /etc/network/interfaces.d/*
+Create another QinQ zone named `qinqzone2` with service VLAN 30
+----
+ID: qinqzone2
+Bridge: vmbr0
+Service VLAN: 30
----
-Create an QinQ zone named `qinqzone1' with service VLAN 20
+Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
+`qinqzone1` zone.
----
-id: qinqzone1
-bridge: vmbr0
-service vlan: 20
+ID: qinqvnet1
+Zone: qinqzone1
+Tag: 100
----
-Create another QinQ zone named `qinqzone2' with service VLAN 30
+Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
----
-id: qinqzone2
-bridge: vmbr0
-service vlan: 30
+ID: qinqvnet2
+Zone: qinqzone2
+Tag: 100
----
-Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
-created `qinqzone1' zone.
+Apply the configuration on the main SDN web-interface panel to create VNets
+locally on each node.
-----
-id: myvnet1
-zone: qinqzone1
-tag: 100
-----
+Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
+interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
+`qinqvnet2`.
-Create a `myvnet2' with customer VLAN-id 100 on the previously created
-`qinqzone2' zone.
+Inside the VM, configure the IP addresses of the interfaces, for example via
+`/etc/network/interfaces`:
----
-id: myvnet2
-zone: qinqzone2
-tag: 100
+auto eth0
+iface eth0 inet static
+ address 10.0.3.101/24
----
+// TODO: systemd-network example
+Configure all four VMs to have IP addresses from the '10.0.3.101' to
+'10.0.3.104' range.
-Apply the configuration on the main SDN web-interface panel to create VNets
-locally on each nodes.
+Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
+between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
+'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
-Use the following network configuration for this VM:
+[[pvesdn_setup_example_vxlan]]
+VXLAN Setup Example
+~~~~~~~~~~~~~~~~~~~
+
+The example assumes a cluster with three nodes, with the node IP addresses
+192.168.0.1, 192.168.0.2 and 192.168.0.3.
+
+Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
+peer address list. Use the default MTU of 1450 or configure accordingly.
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.100/24
+ID: myvxlanzone
+Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
-`myvnet1' as vm1.
-
-Use the following network configuration for this VM:
+Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
+previously.
----
-auto eth0
-iface eth0 inet static
- address 10.0.3.101/24
+ID: vxvnet1
+Zone: myvxlanzone
+Tag: 100000
----
-Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
-`myvnet2'.
+Apply the configuration on the main SDN web-interface panel to create VNets
+locally on each nodes.
-Use the following network configuration for this VM:
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
+
+Use the following network configuration for this VM (note the lower MTU).
----
auto eth0
iface eth0 inet static
- address 10.0.3.102/24
+ address 10.0.3.100/24
+ mtu 1450
----
-Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
-`myvnet2' as vm3.
+Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
+`vxvnet1` as vm1.
Use the following network configuration for this VM:
----
auto eth0
iface eth0 inet static
- address 10.0.3.103/24
+ address 10.0.3.101/24
+ mtu 1450
----
-Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
-between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
-or 'vm4', as they are on a different zone with different service-vlan.
+Then, you should be able to ping between between 'vm1' and 'vm2'.
-[[pvesdn_setup_example_vxlan]]
-VXLAN Setup Example
-~~~~~~~~~~~~~~~~~~~
+[[pvesdn_setup_example_evpn]]
+EVPN Setup Example
+~~~~~~~~~~~~~~~~~~
-TIP: While we show plain configuration content here, almost everything should
-be configurable using the web-interface only.
+The example assumes a cluster with three nodes (node1, node2, node3) with IP
+addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
-node1: /etc/network/interfaces
+Create an EVPN controller, using a private ASN number and the above node
+addresses as peers.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.1/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: myevpnctl
+ASN#: 65000
+Peers: 192.168.0.1,192.168.0.2,192.168.0.3
----
-node2: /etc/network/interfaces
+Create an EVPN zone named `myevpnzone`, assign the previously created
+EVPN-controller and define 'node1' and 'node2' as exit nodes.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.2/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
-
-source /etc/network/interfaces.d/*
+ID: myevpnzone
+VRF VXLAN Tag: 10000
+Controller: myevpnctl
+MTU: 1450
+VNet MAC Address: 32:F4:05:FE:6C:0A
+Exit Nodes: node1,node2
----
-node3: /etc/network/interfaces
+Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.3/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+ID: myvnet1
+Zone: myevpnzone
+Tag: 11000
+----
+
+Create a subnet on `myvnet1`:
-source /etc/network/interfaces.d/*
+----
+Subnet: 10.0.1.0/24
+Gateway: 10.0.1.1
----
-Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
-50 bytes of the VXLAN header can fit. Add all previously configured IPs from
-the nodes as peer address list.
+Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
----
-id: myvxlanzone
-peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
-mtu: 1450
+ID: myvnet2
+Zone: myevpnzone
+Tag: 12000
----
-Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
-previously.
+Create a different subnet on `myvnet2``:
----
-id: myvnet1
-zone: myvxlanzone
-tag: 100000
+Subnet: 10.0.2.0/24
+Gateway: 10.0.2.1
----
-Apply the configuration on the main SDN web-interface panel to create VNets
-locally on each nodes.
+Apply the configuration from the main SDN web-interface panel to create VNets
+locally on each node and generate the FRR configuration.
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
+Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
-Use the following network configuration for this VM, note the lower MTU here.
+Use the following network configuration for 'vm1':
----
auto eth0
iface eth0 inet static
- address 10.0.3.100/24
- mtu 1450
+ address 10.0.1.100/24
+ gateway 10.0.1.1
+ mtu 1450
----
-Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
-`myvnet1' as vm1.
+Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
+`myvnet2`.
-Use the following network configuration for this VM:
+Use the following network configuration for 'vm2':
----
auto eth0
iface eth0 inet static
- address 10.0.3.101/24
- mtu 1450
+ address 10.0.2.100/24
+ gateway 10.0.2.1
+ mtu 1450
----
-Then, you should be able to ping between between 'vm1' and 'vm2'.
+Now you should be able to ping vm2 from vm1, and vm1 from vm2.
-[[pvesdn_setup_example_evpn]]
-EVPN Setup Example
-~~~~~~~~~~~~~~~~~~
+If you ping an external IP from 'vm2' on the non-gateway node3, the packet
+will go to the configured 'myvnet2' gateway, then will be routed to the exit
+nodes ('node1' or 'node2') and from there it will leave those nodes over the
+default gateway configured on node1 or node2.
-node1: /etc/network/interfaces
+NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
+networks to node1 and node2 on your external gateway, so that the public network
+can reply back.
-----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.1/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
+and 10.0.2.0/24 in this example), will be announced dynamically.
-source /etc/network/interfaces.d/*
-----
-node2: /etc/network/interfaces
+[[pvesdn_notes]]
+Notes
+-----
-----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.2/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+Multiple EVPN Exit Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~
-source /etc/network/interfaces.d/*
-----
+If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
+Reverse Path Filter) option, because packets can arrive at one node but go out
+from another node.
-node3: /etc/network/interfaces
+Add the following to `/etc/sysctl.conf`:
-----
-auto vmbr0
-iface vmbr0 inet static
- address 192.168.0.3/24
- gateway 192.168.0.254
- bridge-ports eno1
- bridge-stp off
- bridge-fd 0
- mtu 1500
+-----
+net.ipv4.conf.default.rp_filter=0
+net.ipv4.conf.all.rp_filter=0
+-----
-source /etc/network/interfaces.d/*
-----
+VXLAN IPSEC Encryption
+~~~~~~~~~~~~~~~~~~~~~~
-Create a EVPN controller, using a private ASN number and above node addreesses
-as peers.
+To add IPSEC encryption on top of a VXLAN, this example shows how to use
+`strongswan`.
-----
-id: myevpnctl
-asn: 65000
-peers: 192.168.0.1,192.168.0.2,192.168.0.3
-----
+You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
+IPv6 to handle encryption.
-Create an EVPN zone named `myevpnzone' using the previously created
-EVPN-controller Define 'node1' and 'node2' as exit nodes.
+So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
++ 50 (VXLAN) == 1500).
+Install strongswan on the host.
----
-id: myevpnzone
-vrf vxlan tag: 10000
-controller: myevpnctl
-mtu: 1450
-exitnodes: node1,node2
+apt install strongswan
----
-Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
-----
-id: myvnet1
-zone: myevpnzone
-tag: 11000
-mac address: 8C:73:B2:7B:F9:60 #random generate mac address
-----
+Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
+the VXLAN UDP port '4789'.
-Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway
-----
-id: 10.0.1.0/24
-gateway: 10.0.1.1
----
+conn %default
+ ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
+ esp=aes256-sha1!
+ leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
-Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
-different IPv4 CIDR network and a different random MAC address than `myvnet1'.
+conn output
+ rightsubnet=%dynamic[udp/4789]
+ right=%any
+ type=transport
+ authby=psk
+ auto=route
+conn input
+ leftsubnet=%dynamic[udp/4789]
+ type=transport
+ authby=psk
+ auto=route
----
-id: myvnet2
-zone: myevpnzone
-tag: 12000
-mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet
-----
-
-Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway
-----
-id: 10.0.2.0/24
-gateway: 10.0.2.1
-----
-
-
-Apply the configuration on the main SDN web-interface panel to create VNets
-locally on each nodes and generate the FRR config.
-
-Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
-
-Use the following network configuration for this VM:
+Generate a pre-shared key with:
----
-auto eth0
-iface eth0 inet static
- address 10.0.1.100/24
- gateway 10.0.1.1 #this is the ip of the vnet1
- mtu 1450
+openssl rand -base64 128
----
-Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
-`myvnet2'.
-
-Use the following network configuration for this VM:
+and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
----
-auto eth0
-iface eth0 inet static
- address 10.0.2.100/24
- gateway 10.0.2.1 #this is the ip of the vnet2
- mtu 1450
+: PSK <generatedbase64key>
----
-
-Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
-
-If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
-will go to the configured 'myvnet2' gateway, then will be routed to the exit
-nodes ('node1' or 'node2') and from there it will leave those nodes over the
-default gateway configured on node1 or node2.
-
-NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
-'10.0.2.0/24' network to node1, node2 on your external gateway, so that the
-public network can reply back.
-
-If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
-and 10.0.2.0/24 in this example), will be announced dynamically.
+Copy the PSK and the configuration to all nodes participating in the VXLAN network.
projects / pve-docs.git / blobdiff