[[chapter_pvesdn]] Software Defined Network ======================== ifndef::manvolnum[] :pve-toplevel: endif::manvolnum[] The SDN feature allow to create virtual networks (vnets) at datacenter level. To enable SDN feature, you need to install "libpve-network-perl" package ---- apt install libpve-network-perl ---- A vnet is a bridge with a vlan or vxlan tag. The vnets are deployed locally on each node after configuration commit at datacenter level. You need to have "ifupdown2" package installed on each node to manage local configuration reloading. ---- apt install ifupdown2 ---- Main configuration ------------------ The configuration is done at datacenter level. The sdn feature have 4 main sections for the configuration * SDN * Zones * Vnets * Controller SDN ~~~ [thumbnail="screenshot/gui-sdn-status.png"] This is the Main panel, where you can see deployment of zones on differents nodes. They are an "apply" button, to push && reload local configuration on differents nodes. Zones ~~~~~ [thumbnail="screenshot/gui-sdn-zone.png"] A zone will defined the kind of virtual network you want to defined. it can be * vlan * QinQ (stacked vlan) * vxlan (layer2 vxlan) * bgp-evpn (vxlan with layer3 routing) You can restrict a zone to specific nodes. It's also possible to add permissions on a zone, to restrict user to use only a specific zone and the vnets in this zone Vnets ~~~~~ [thumbnail="screenshot/gui-sdn-vnet-evpn.png"] A vnet is a bridge that will be deployed locally on the node, for vm communication. (Like a classic vmbrX). Vnet properties are: * ID: a 8 characters ID * Alias: Optionnal bigger name * Zone: The associated zone of the vnet * Tag: unique vlan or vxlan id * ipv4: an anycast ipv4 address (same bridge ip deployed on each node), for bgp-evpn routing only * ipv6: an anycast ipv6 address (same bridge ip deployed on each node), for bgp-evpn routing only Controllers ~~~~~~~~~~~ [thumbnail="screenshot/gui-sdn-controller.png"] Some zone plugins (Currently : bgp-evpn only), need an external controller to manage the vnets control-plane. Zones Plugins ------------- common zone options: * nodes: restrict deploy of the vnets of theses nodes only Vlan ~~~~~ [thumbnail="screenshot/gui-sdn-zone-vlan.png"] This is the most simple plugin, it'll reuse an existing local bridge or ovs, and manage vlan on it. The benefit of using sdn module, is that you can create different zones with specific vnets vlan tag, and restrict your customers on their zones. specific qinq configuration options: * bridge: a local vlan-aware bridge or ovs switch already configured on each local node QinQ ~~~~~ [thumbnail="screenshot/gui-sdn-zone-qinq.png"] QinQ is stacked vlan. you have the first vlan tag defined on the zone (service-vlan), and the second vlan tag defined on the vnets Your physical network switchs need to support stacked vlans ! specific qinq configuration options: * bridge: a local vlan-aware bridge already configured on each local node * service vlan: The main vlan tag of this zone * mtu: you need 4 more bytes for the double tag vlan. You can reduce the mtu to 1496 if you physical interface mtu is 1500. Vxlan ~~~~~ [thumbnail="screenshot/gui-sdn-zone-vxlan.png"] The vxlan plugin will established vxlan tunnel (overlay) on top of an existing network (underlay). you can for example, create a private ipv4 vxlan network on top of public internet network nodes. This is a layer2 tunnel only, no routing between different vnets is possible. Each vnet will have a specific vxlan id ( 1 - 16777215 ) Specific evpn configuration options: * peers address list: an ip list of all nodes where you want to communicate (could be also external nodes) * mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower than the outgoing physical interface. evpn ~~~~ [thumbnail="screenshot/gui-sdn-zone-evpn.png"] This is the most complex plugin. BGP-evpn allow to create routable layer3 network. The vnet of evpn can have an anycast ip address/mac address. The bridge ip is the same on each node, then vm can use as gateway. The routing is working only across vnets of a specific zone through a vrf. Specific evpn configuration options: * vrf vxlan tag: This is a vxlan-id used for routing interconnect between vnets, it must be different than vxlan-id of vnets * controller: an evpn need to be defined first (see controller plugins section) * mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower than the outgoing physical interface. Controllers Plugins ------------------- evpn ~~~~ [thumbnail="screenshot/gui-sdn-controller-evpn.png"] For bgp-evpn, we need a controller to manage the control plane. The software controller is "frr" router. You need to install it on each node where you want to deploy the evpn zone. ---- apt install frr ---- configuration options: *asn: a unique bgp asn number. It's recommended to use private asn number (64512 – 65534, 4200000000 – 4294967294) *peers: an ip list of all nodes where you want to communicate (could be also external nodes or route reflectors servers) If you want to route traffic from the sdn bgp-evpn network to external world: * gateway-nodes: The proxmox nodes from where the bgp-evpn traffic will exit to external through the nodes default gateway If you want that gateway nodes don't use the default gateway, but for example, sent traffic to external bgp routers * gateway-external-peers: 192.168.0.253,192.168.0.254 Local deployment Monitoring --------------------------- [thumbnail="screenshot/gui-sdn-local-status.png"] After apply configuration on the main sdn section, the local configuration is generated locally on each node, in /etc/network/interfaces.d/sdn, and reloaded. You can monitor the status of local zones && vnets through the main tree. Vlan setup example ------------------ node1: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet manual bridge-ports eno1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 #management ip on vlan100 auto vmbr0.100 iface vmbr0.100 inet static address 192.168.0.1/24 source /etc/network/interfaces.d/* ---- node2: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet manual bridge-ports eno1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 #management ip on vlan100 auto vmbr0.100 iface vmbr0.100 inet static address 192.168.0.2/24 source /etc/network/interfaces.d/* ---- create an vlan zone ---- id: mylanzone bridge: vmbr0 ---- create a vnet1 with vlan-id 10 ---- id: myvnet1 zone: myvlanzone tag: 10 ---- Apply the configuration on the main sdn section, to create vnets locally on each nodes, and generate frr config. create a vm1, with 1 nic on vnet1 on node1 ---- auto eth0 iface eth0 inet static address 10.0.3.100/24 ---- create a vm2, with 1 nic on vnet1 on node2 ---- auto eth0 iface eth0 inet static address 10.0.3.101/24 ---- Then, you should be able to ping between between vm1 && vm2 QinQ setup example ------------------ node1: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet manual bridge-ports eno1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 #management ip on vlan100 auto vmbr0.100 iface vmbr0.100 inet static address 192.168.0.1/24 source /etc/network/interfaces.d/* ---- node2: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet manual bridge-ports eno1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 #management ip on vlan100 auto vmbr0.100 iface vmbr0.100 inet static address 192.168.0.2/24 source /etc/network/interfaces.d/* ---- create an qinq zone1 with service vlan 20 ---- id: qinqzone1 bridge: vmbr0 service vlan: 20 ---- create an qinq zone2 with service vlan 30 ---- id: qinqzone2 bridge: vmbr0 service vlan: 30 ---- create a vnet1 with customer vlan-id 100 on qinqzone1 ---- id: myvnet1 zone: qinqzone1 tag: 100 ---- create a vnet2 with customer vlan-id 100 on qinqzone2 ---- id: myvnet2 zone: qinqzone1 tag: 100 ---- Apply the configuration on the main sdn section, to create vnets locally on each nodes, and generate frr config. create a vm1, with 1 nic on vnet1 on node1 ---- auto eth0 iface eth0 inet static address 10.0.3.100/24 ---- create a vm2, with 1 nic on vnet1 on node2 ---- auto eth0 iface eth0 inet static address 10.0.3.101/24 ---- create a vm3, with 1 nic on vnet2 on node1 ---- auto eth0 iface eth0 inet static address 10.0.3.102/24 ---- create a vm4, with 1 nic on vnet2 on node2 ---- auto eth0 iface eth0 inet static address 10.0.3.103/24 ---- Then, you should be able to ping between between vm1 && vm2 vm3 && vm4 could ping together but vm1 && vm2 couldn't ping vm3 && vm4, as it's a different zone, with different service vlan Vxlan setup example ------------------- node1: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet static address 192.168.0.1/24 gateway 192.168.0.254 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 1500 source /etc/network/interfaces.d/* ---- node2: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet static address 192.168.0.2/24 gateway 192.168.0.254 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 1500 source /etc/network/interfaces.d/* ---- node3: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet static address 192.168.0.3/24 gateway 192.168.0.254 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 1500 source /etc/network/interfaces.d/* ---- create an vxlan zone ---- id: myvxlanzone peers address list: 192.168.0.1,192.168.0.2,192.168.0.3 mtu: 1450 ---- create first vnet ---- id: myvnet1 zone: myvxlanzone tag: 100000 ---- Apply the configuration on the main sdn section, to create vnets locally on each nodes, and generate frr config. create a vm1, with 1 nic on vnet1 on node2 ---- auto eth0 iface eth0 inet static address 10.0.3.100/24 mtu 1450 ---- create a vm2, with 1 nic on vnet1 on node3 ---- auto eth0 iface eth0 inet static address 10.0.3.101/24 mtu 1450 ---- Then, you should be able to ping between between vm1 && vm2 EVPN setup example ------------------ node1: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet static address 192.168.0.1/24 gateway 192.168.0.254 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 1500 source /etc/network/interfaces.d/* ---- node2: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet static address 192.168.0.2/24 gateway 192.168.0.254 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 1500 source /etc/network/interfaces.d/* ---- node3: /etc/network/interfaces ---- auto vmbr0 iface vmbr0 inet static address 192.168.0.3/24 gateway 192.168.0.254 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 1500 source /etc/network/interfaces.d/* ---- create a evpn controller ---- id: myevpnctl asn: 65000 peers: 192.168.0.1,192.168.0.2,192.168.0.3 gateway nodes: node1,node2 ---- create an evpn zone ---- id: myevpnzone vrf vxlan tag: 10000 controller: myevpnctl mtu: 1450 ---- create first vnet ---- id: myvnet1 zone: myevpnzone tag: 11000 ipv4: 10.0.1.1/24 mac address: 8C:73:B2:7B:F9:60 #random generate mac addres ---- create second vnet ---- id: myvnet2 zone: myevpnzone tag: 12000 ipv4: 10.0.2.1/24 mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet ---- Apply the configuration on the main sdn section, to create vnets locally on each nodes, and generate frr config. create a vm1, with 1 nic on vnet1 on node2 ---- auto eth0 iface eth0 inet static address 10.0.1.100/24 gateway 10.0.1.1 #this is the ip of the vnet1 mtu 1450 ---- create a vm2, with 1 nic on vnet2 on node3 ---- auto eth0 iface eth0 inet static address 10.0.2.100/24 gateway 10.0.2.1 #this is the ip of the vnet2 mtu 1450 ---- Then, you should be able to ping vm2 from vm1, and vm1 from vm2. from vm2 on node3, if you ping an external ip, the packet will go to the vnet2 gateway, then will be routed to gateway nodes (node1 or node2) then the packet will be routed to the node1 or node2 default gw. Of course you need to add reverse routes to 10.0.1.0/24 && 10.0.2.0/24 to node1,node2 on your external gateway. If you have configured an external bgp router, the bgp-evpn routes (10.0.1.0/24 && 10.0.2.0/24), will be announced dynamically.