add sdn documentation

[pve-docs.git] / pvesdn.adoc

[[chapter_pvesdn]]
Software Defined Network
========================
ifndef::manvolnum[]
:pve-toplevel:
endif::manvolnum[]

The SDN feature allow to create virtual networks (vnets)
at datacenter level.

To enable SDN feature, you need to install "libpve-network-perl" package

----
apt install libpve-network-perl
----

A vnet is a bridge with a vlan or vxlan tag.

The vnets are deployed locally on each node after configuration
commit at datacenter level.

You need to have "ifupdown2" package installed on each node to manage local
configuration reloading.

----
apt install ifupdown2
----

Main configuration
------------------

The configuration is done at datacenter level.

The sdn feature have 4 main sections for the configuration

* SDN 

* Zones

* Vnets

* Controller


SDN
~~~

[thumbnail="screenshot/gui-sdn-status.png"]

This is the Main panel, where you can see deployment of zones on differents nodes.

They are an "apply" button, to push && reload local configuration on differents nodes.


Zones
~~~~~

[thumbnail="screenshot/gui-sdn-zone.png"]

A zone will defined the kind of virtual network you want to defined.

it can be

* vlan

* QinQ (stacked vlan)

* vxlan (layer2 vxlan)

* bgp-evpn  (vxlan with layer3 routing)

You can restrict a zone to specific nodes.

It's also possible to add permissions on a zone, to restrict user
to use only a specific zone and the vnets in this zone

Vnets
~~~~~

[thumbnail="screenshot/gui-sdn-vnet-evpn.png"]

A vnet is a bridge that will be deployed locally on the node,
for vm communication. (Like a classic vmbrX).

Vnet properties are:

* ID: a 8 characters ID

* Alias: Optionnal bigger name

* Zone: The associated zone of the vnet

* Tag: unique vlan or vxlan id

* ipv4: an anycast ipv4 address (same bridge ip deployed on each node), for bgp-evpn routing only 

* ipv6: an anycast ipv6 address (same bridge ip deployed on each node), for bgp-evpn routing only


Controllers
~~~~~~~~~~~

[thumbnail="screenshot/gui-sdn-controller.png"]

Some zone plugins (Currently : bgp-evpn only),
need an external controller to manage the vnets control-plane.



Zones Plugins
-------------
common zone options:

* nodes: restrict deploy of the vnets of theses nodes only


Vlan
~~~~~

[thumbnail="screenshot/gui-sdn-zone-vlan.png"]

This is the most simple plugin, it'll reuse an existing local bridge or ovs,
and manage vlan on it.
The benefit of using sdn module, is that you can create different zones with specific
vnets vlan tag, and restrict your customers on their zones.

specific qinq configuration options:

* bridge: a local vlan-aware bridge or ovs switch already configured on each local node

QinQ
~~~~~

[thumbnail="screenshot/gui-sdn-zone-qinq.png"]

QinQ is stacked vlan.
you have the first vlan tag defined on the zone (service-vlan), and
the second vlan tag defined on the vnets

Your physical network switchs need to support stacked vlans !

specific qinq configuration options:

* bridge: a local vlan-aware bridge already configured on each local node
* service vlan: The main vlan tag of this zone
* mtu: you need 4 more bytes for the double tag vlan. 
       You can reduce the mtu to 1496 if you physical interface mtu is 1500.

Vxlan
~~~~~

[thumbnail="screenshot/gui-sdn-zone-vxlan.png"]

The vxlan plugin will established vxlan tunnel (overlay) on top of an existing network (underlay).
you can for example, create a private ipv4 vxlan network on top of public internet network nodes.
This is a layer2 tunnel only, no routing between different vnets is possible.

Each vnet will have a specific vxlan id ( 1 - 16777215 )


Specific evpn configuration options:

* peers address list: an ip list of all nodes where you want to communicate (could be also external nodes)

* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
       than the outgoing physical interface.

evpn
~~~~

[thumbnail="screenshot/gui-sdn-zone-evpn.png"]

This is the most complex plugin.

BGP-evpn allow to create routable layer3 network.
The vnet of evpn can have an anycast ip address/mac address.
The bridge ip is the same on each node, then vm can use
as gateway.
The routing is working only across vnets of a specific zone through a vrf.

Specific evpn configuration options:

* vrf vxlan tag: This is a vxlan-id used for routing interconnect between vnets,
                 it must be different than vxlan-id of vnets

* controller: an evpn need to be defined first (see controller plugins section)

* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
       than the outgoing physical interface.


Controllers Plugins
-------------------

evpn
~~~~

[thumbnail="screenshot/gui-sdn-controller-evpn.png"]

For bgp-evpn, we need a controller to manage the control plane.
The software controller is "frr" router.
You need to install it on each node where you want to deploy the evpn zone.

----
apt install frr
----

configuration options:

*asn: a unique bgp asn number. 
      It's recommended to use private asn number (64512 – 65534, 4200000000 – 4294967294)

*peers: an ip list of all nodes where you want to communicate (could be also external nodes or route reflectors servers)

If you want to route traffic from the sdn bgp-evpn network to external world:

* gateway-nodes: The proxmox nodes from where the bgp-evpn traffic will exit to external through the nodes default gateway

If you want that gateway nodes don't use the default gateway, but for example, sent traffic to external bgp routers

* gateway-external-peers: 192.168.0.253,192.168.0.254


Local deployment Monitoring
---------------------------

[thumbnail="screenshot/gui-sdn-local-status.png"]

After apply configuration on the main sdn section,
the local configuration is generated locally on each node,
in /etc/network/interfaces.d/sdn, and reloaded.

You can monitor the status of local zones && vnets through the main tree.



Vlan setup example
------------------
node1: /etc/network/interfaces
----
auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
	address 192.168.0.1/24

source /etc/network/interfaces.d/*

----

node2: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
	address 192.168.0.2/24

source /etc/network/interfaces.d/*
----

create an vlan zone

----
id: mylanzone
bridge: vmbr0
----

create a vnet1 with vlan-id 10

----
id: myvnet1
zone: myvlanzone
tag: 10
----

Apply the configuration on the main sdn section, to create vnets locally on each nodes,
and generate frr config.


create a vm1, with 1 nic on vnet1 on node1

----
auto eth0
iface eth0 inet static
        address 10.0.3.100/24
----

create a vm2, with 1 nic on vnet1 on node2
----
auto eth0
iface eth0 inet static
        address 10.0.3.101/24
----

Then, you should be able to ping between between vm1 && vm2


QinQ setup example
------------------
node1: /etc/network/interfaces
----
auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
	address 192.168.0.1/24

source /etc/network/interfaces.d/*
----

node2: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
	address 192.168.0.2/24

source /etc/network/interfaces.d/*
----

create an qinq zone1 with service vlan 20

----
id: qinqzone1
bridge: vmbr0
service vlan: 20
----

create an qinq zone2 with service vlan 30

----
id: qinqzone2
bridge: vmbr0
service vlan: 30
----

create a vnet1 with customer vlan-id 100 on qinqzone1

----
id: myvnet1
zone: qinqzone1
tag: 100
----

create a vnet2 with customer vlan-id 100 on qinqzone2

----
id: myvnet2
zone: qinqzone1
tag: 100
----

Apply the configuration on the main sdn section, to create vnets locally on each nodes,
and generate frr config.


create a vm1, with 1 nic on vnet1 on node1

----
auto eth0
iface eth0 inet static
        address 10.0.3.100/24
----

create a vm2, with 1 nic on vnet1 on node2
----
auto eth0
iface eth0 inet static
        address 10.0.3.101/24
----

create a vm3, with 1 nic on vnet2 on node1

----
auto eth0
iface eth0 inet static
        address 10.0.3.102/24
----

create a vm4, with 1 nic on vnet2 on node2
----
auto eth0
iface eth0 inet static
        address 10.0.3.103/24
----

Then, you should be able to ping between between vm1 && vm2
vm3 && vm4 could ping together

but vm1 && vm2  couldn't ping vm3 && vm4,
as it's a different zone, with different service vlan


Vxlan setup example
-------------------
node1: /etc/network/interfaces
----
auto vmbr0
iface vmbr0 inet static
	address 192.168.0.1/24
	gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	mtu 1500

source /etc/network/interfaces.d/*
----

node2: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet static
	address 192.168.0.2/24
	gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	mtu 1500

source /etc/network/interfaces.d/*
----

node3: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet static
	address 192.168.0.3/24
	gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	mtu 1500

source /etc/network/interfaces.d/*
----

create an vxlan zone

----
id: myvxlanzone
peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
mtu: 1450
----

create first vnet

----
id: myvnet1
zone: myvxlanzone
tag: 100000
----

Apply the configuration on the main sdn section, to create vnets locally on each nodes,
and generate frr config.


create a vm1, with 1 nic on vnet1 on node2

----
auto eth0
iface eth0 inet static
        address 10.0.3.100/24
        mtu 1450
----

create a vm2, with 1 nic on vnet1 on node3
----
auto eth0
iface eth0 inet static
        address 10.0.3.101/24
        mtu 1450
----

Then, you should be able to ping between between vm1 && vm2



EVPN setup example
------------------
node1: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet static
	address 192.168.0.1/24
	gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	mtu 1500

source /etc/network/interfaces.d/*
----

node2: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet static
	address 192.168.0.2/24
	gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	mtu 1500

source /etc/network/interfaces.d/*
----

node3: /etc/network/interfaces

----
auto vmbr0
iface vmbr0 inet static
	address 192.168.0.3/24
	gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
	mtu 1500

source /etc/network/interfaces.d/*
----

create a evpn controller

----
id: myevpnctl
asn: 65000
peers: 192.168.0.1,192.168.0.2,192.168.0.3
gateway nodes: node1,node2
----

create an evpn zone

----
id: myevpnzone
vrf vxlan tag: 10000
controller: myevpnctl
mtu: 1450
----

create first vnet

----
id: myvnet1
zone: myevpnzone
tag: 11000
ipv4: 10.0.1.1/24
mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
----

create second vnet

----
id: myvnet2
zone: myevpnzone
tag: 12000
ipv4: 10.0.2.1/24
mac address: 8C:73:B2:7B:F9:61  #random mac, need to be different on each vnet
----

Apply the configuration on the main sdn section, to create vnets locally on each nodes,
and generate frr config.



create a vm1, with 1 nic on vnet1 on node2

----
auto eth0
iface eth0 inet static
        address 10.0.1.100/24
        gateway 10.0.1.1   #this is the ip of the vnet1 
        mtu 1450
----

create a vm2, with 1 nic on vnet2 on node3
----
auto eth0
iface eth0 inet static
        address 10.0.2.100/24
        gateway 10.0.2.1   #this is the ip of the vnet2
        mtu 1450
----


Then, you should be able to ping vm2 from vm1, and vm1 from vm2.

from vm2 on node3, if you ping an external ip, the packet will go 
to the vnet2 gateway, then will be routed to gateway nodes (node1 or node2)
then the packet will be routed to the node1 or node2 default gw.

Of course you need to add reverse routes to 10.0.1.0/24 && 10.0.2.0/24 to node1,node2 on your external gateway.

If you have configured an external bgp router, the bgp-evpn routes (10.0.1.0/24 && 10.0.2.0/24),
will be announced dynamically.