]> git.proxmox.com Git - pve-docs.git/blame - pvesdn.adoc
sdn: overall language and consistency rework
[pve-docs.git] / pvesdn.adoc
CommitLineData
1556b768 1[[chapter_pvesdn]]
3c126a21 2Software-Defined Network
1556b768
AD
3========================
4ifndef::manvolnum[]
5:pve-toplevel:
6endif::manvolnum[]
7
3c126a21 8The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
5899fa0e 9virtual networks (VNets) at the datacenter level.
1556b768 10
ee6e18c4 11WARNING: SDN is currently an **experimental feature** in {pve}. This
5899fa0e 12documentation for it is also still under development. Ask on our
ee6e18c4
TL
13xref:getting_help[mailing lists or in the forum] for questions and feedback.
14
15
4e652aba 16[[pvesdn_installation]]
ee6e18c4
TL
17Installation
18------------
19
3c126a21 20To enable the experimental Software-Defined Network (SDN) integration, you need
bcb72b25 21to install the `libpve-network-perl` package on every node:
1556b768
AD
22
23----
684db7e3 24apt update
bcb72b25 25apt install libpve-network-perl
1556b768
AD
26----
27
bcb72b25
SL
28NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
29default. If you originally installed your system with an older version, you need
30to explicitly install the `ifupdown2` package.
5899fa0e 31
bcb72b25 32After installation, you need to add the following line to the end of the
5899fa0e
DW
33`/etc/network/interfaces` configuration file, so that the SDN configuration gets
34included and activated.
1556b768 35
a6af82c7
AD
36----
37source /etc/network/interfaces.d/*
38----
a6af82c7
AD
39
40
bcb72b25
SL
41[[pvesdn_overview]]
42Overview
43--------
ee6e18c4 44
5899fa0e
DW
45The {pve} SDN allows for separation and fine-grained control of virtual guest
46networks, using flexible, software-controlled configurations.
ee6e18c4 47
bcb72b25
SL
48Separation is managed through *zones*, virtual networks (*VNets*), and
49*subnets*. A zone is its own virtually separated network area. A VNet is a
50virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
51
52Depending on the type of the zone, the network behaves differently and offers
53specific features, advantages, and limitations.
54
55Use cases for SDN range from an isolated private network on each individual node
56to complex overlay networks across multiple PVE clusters on different locations.
ee6e18c4 57
bcb72b25
SL
58After configuring an VNet in the cluster-wide datacenter SDN administration
59interface, it is available as a common Linux bridge, locally on each node, to be
60assigned to VMs and Containers.
ee6e18c4 61
bcb72b25
SL
62
63[[pvesdn_main_configuration]]
5899fa0e 64Main Configuration
a6af82c7 65~~~~~~~~~~~~~~~~~~
1556b768 66
bcb72b25
SL
67Configuration is done at the web UI at datacenter level and is saved in files
68located in the shared configuration file system at `/etc/pve/sdn`.
1556b768 69
bcb72b25 70On the web interface, SDN features the following sections:
1556b768 71
bcb72b25
SL
72* SDN:: Here you get an overview of the current active SDN state, and you can
73 apply all pending changes to the whole cluster.
1556b768 74
bcb72b25
SL
75* xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
76 network zones
a6af82c7 77
bcb72b25
SL
78* xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and
79 manage subnets
1556b768 80
bcb72b25
SL
81The Options category allows adding and managing additional services to be used
82in your SDN setup.
1556b768 83
bcb72b25
SL
84* xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
85 in complex setups
a6af82c7 86
bcb72b25
SL
87* xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
88 guests
a6af82c7 89
bcb72b25
SL
90* xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
91 virtual guests' hostname and IP
5899fa0e 92 addresses
1556b768 93
4e652aba 94[[pvesdn_config_main_sdn]]
1556b768
AD
95SDN
96~~~
97
5899fa0e
DW
98This is the main status panel. Here you can see the deployment status of zones
99on different nodes.
1556b768 100
bcb72b25 101Pressing the 'Apply' button reloads the local configuration on all cluster
5899fa0e 102nodes.
1556b768 103
4e652aba 104[[pvesdn_config_zone]]
1556b768 105Zones
a6af82c7 106-----
1556b768 107
bcb72b25 108A zone defines a virtually separated network. Zones are restricted to
5899fa0e
DW
109specific nodes and assigned permissions, in order to restrict users to a certain
110zone and its contained VNets.
1556b768 111
5899fa0e 112Different technologies can be used for separation:
1556b768 113
bcb72b25
SL
114* Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
115
5899fa0e 116* VLAN: Virtual LANs are the classic method of subdividing a LAN
1556b768 117
5899fa0e 118* QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
1556b768 119
bcb72b25 120* VXLAN: Layer 2 VXLAN network via a UDP tunnel
a6af82c7 121
bcb72b25 122* EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
1556b768 123
1556b768 124
bcb72b25
SL
125[[pvesdn_config_common_options]]
126Common Options
a6af82c7 127~~~~~~~~~~~~~~
1556b768 128
5899fa0e 129The following options are available for all zone types:
3093e62d 130
bcb72b25 131Nodes:: The nodes which the zone and associated VNets should be deployed on.
9e773815 132
bcb72b25
SL
133IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
134 zone. Optional, defaults to `pve`.
1556b768 135
bcb72b25 136DNS:: DNS API server. Optional.
1556b768 137
bcb72b25 138ReverseDNS:: Reverse DNS API server. Optional.
1556b768 139
bcb72b25
SL
140DNSZone:: DNS domain name. Used to register hostnames, such as
141 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
1556b768
AD
142
143
a6af82c7
AD
144[[pvesdn_zone_plugin_simple]]
145Simple Zones
146~~~~~~~~~~~~
1556b768 147
bcb72b25
SL
148This is the simplest plugin. It will create an isolated VNet bridge. This
149bridge is not linked to a physical interface, and VM traffic is only local on
150each the node.
151It can be used in NAT or routed setups.
152
1556b768 153
4e652aba 154[[pvesdn_zone_plugin_vlan]]
ee6e18c4
TL
155VLAN Zones
156~~~~~~~~~~
1556b768 157
bcb72b25
SL
158The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
159node's physical interface. It uses VLAN tagging defined in the VNet to isolate
160the network segments. This allows connectivity of VMs between different nodes.
161
162VLAN zone configuration options:
1556b768 163
bcb72b25
SL
164Bridge:: The local bridge or OVS switch, already configured on *each* node that
165 allows node-to-node connection.
1556b768 166
1556b768 167
4e652aba 168[[pvesdn_zone_plugin_qinq]]
ee6e18c4
TL
169QinQ Zones
170~~~~~~~~~~
1556b768 171
bcb72b25
SL
172QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
173isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
174whereas the inner VLAN tag is defined by the VNet.
1556b768 175
5899fa0e 176NOTE: Your physical network switches must support stacked VLANs for this
bcb72b25 177configuration.
1556b768 178
bcb72b25 179QinQ zone configuration options:
1556b768 180
bcb72b25 181Bridge:: A local, VLAN-aware bridge that is already configured on each local
5899fa0e 182 node
4e652aba 183
bcb72b25 184Service VLAN:: The main VLAN tag of this zone
4e652aba 185
bcb72b25 186Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
5899fa0e 187 802.1ad service VLAN type.
9a4f3c95 188
bcb72b25 189MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
5899fa0e
DW
190 For example, you must reduce the MTU to `1496` if you physical interface MTU is
191 `1500`.
1556b768 192
bcb72b25 193
4e652aba 194[[pvesdn_zone_plugin_vxlan]]
ee6e18c4
TL
195VXLAN Zones
196~~~~~~~~~~~
1556b768 197
bcb72b25
SL
198The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
199(underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
200datagrams using the default destination port `4789`.
201
202You have to configure the underlay network yourself to enable UDP connectivity
203between all peers.
1556b768 204
bcb72b25
SL
205You can, for example, create a VXLAN overlay network on top of public internet,
206appearing to the VMs as if they share the same local Layer 2 network.
5899fa0e 207
bcb72b25
SL
208WARNING: VXLAN on its own does does not provide any encryption. When joining
209 multiple sites via VXLAN, make sure to establish a secure connection between
210 the site, for example by using a site-to-site VPN.
1556b768 211
bcb72b25 212VXLAN zone configuration options:
1556b768 213
bcb72b25
SL
214Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
215 can be external nodes reachable at this IP address.
216 All nodes in the cluster need to be mentioned here.
4e652aba 217
bcb72b25 218MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
5899fa0e 219 lower than the outgoing physical interface.
1556b768 220
bcb72b25 221
4e652aba 222[[pvesdn_zone_plugin_evpn]]
ee6e18c4
TL
223EVPN Zones
224~~~~~~~~~~
1556b768 225
bcb72b25
SL
226The EVPN zone creates a routable Layer 3 network, capable of spanning across
227multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
228the routing protocol.
1556b768 229
bcb72b25
SL
230The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
231IP is the same on each node, meaning a virtual guest can use this address as
232gateway.
1556b768 233
ee6e18c4
TL
234Routing can work across VNets from different zones through a VRF (Virtual
235Routing and Forwarding) interface.
1556b768 236
bcb72b25 237EVPN zone configuration options:
1556b768 238
bcb72b25 239VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
5899fa0e 240 It must be different than the VXLAN-ID of the VNets.
1556b768 241
bcb72b25 242Controller:: The EVPN-controller to use for this zone. (See controller plugins
5899fa0e 243 section).
1556b768 244
bcb72b25
SL
245VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
246 zone. Will be auto-generated if not defined.
a6af82c7 247
bcb72b25
SL
248Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
249 network, through the real network. The configured nodes will announce a
250 default route in the EVPN network. Optional.
a6af82c7 251
bcb72b25
SL
252Primary Exit Node:: If you use multiple exit nodes, force traffic through this
253 primary exit node, instead of load-balancing on all nodes. Optional but
254 necessary if you want to use SNAT or if your upstream router doesn't support
5899fa0e 255 ECMP.
f4e692a3 256
bcb72b25
SL
257Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
258 service from an exit node. (By default, the exit nodes only allow forwarding
259 traffic between real network and EVPN network). Optional.
f4e692a3 260
bcb72b25
SL
261Advertise Subnets:: Announce the full subnet in the EVPN network.
262 If you have silent VMs/CTs (for example, if you have multiple IPs and the
263 anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
264 able to be reached inside the EVPN network). Optional.
4eb24270 265
bcb72b25
SL
266Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
267 packets. This is required if you use floating IPs in your VMs (IP and MAC
268 addresses are being moved between systems). Optional.
f4e692a3 269
bcb72b25
SL
270Route-target Import:: Allows you to import a list of external EVPN route
271 targets. Used for cross-DC or different EVPN network interconnects. Optional.
4eb24270 272
5899fa0e 273MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
bcb72b25
SL
274 less than the maximal MTU of the outgoing physical interface. Optional,
275 defaults to 1450.
1556b768
AD
276
277
bcb72b25 278[[pvesdn_config_vnets]]
a6af82c7
AD
279VNets
280-----
281
bcb72b25
SL
282After creating a virtual network (VNet) through the SDN GUI, a local network
283interface with the same name is available on each node. To connect a guest to the
284VNet, assign the interface to the guest and set the IP address accordingly.
a6af82c7 285
bcb72b25
SL
286Depending on the zone, these options have different meanings and are explained
287in the respective zone section in this document.
a6af82c7 288
bcb72b25
SL
289WARNING: In the current state, some options may have no effect or won't work in
290certain zones.
a6af82c7 291
bcb72b25
SL
292VNet configuration options:
293
294ID:: An up to 8 character ID to identify a VNet
295
296Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
a6af82c7
AD
297
298Zone:: The associated zone for this VNet
299
5899fa0e 300Tag:: The unique VLAN or VXLAN ID
a6af82c7 301
bcb72b25
SL
302VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
303 in the quest.
304
a6af82c7
AD
305
306[[pvesdn_config_subnet]]
5899fa0e 307Subnets
bcb72b25 308-------
3093e62d 309
bcb72b25
SL
310A subnet define a specific IP range, described by the CIDR network address.
311Each VNet, can have one or more subnets.
1556b768 312
3093e62d 313A subnet can be used to:
a6af82c7 314
5899fa0e
DW
315* Restrict the IP addresses you can define on a specific VNet
316* Assign routes/gateways on a VNet in layer 3 zones
317* Enable SNAT on a VNet in layer 3 zones
318* Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
3093e62d 319* DNS registration through DNS plugins
a6af82c7 320
5899fa0e 321If an IPAM server is associated with the subnet zone, the subnet prefix will be
3093e62d 322automatically registered in the IPAM.
a6af82c7 323
bcb72b25 324Subnet configuration options:
a6af82c7 325
5899fa0e 326ID:: A CIDR network address, for example 10.0.0.0/8
a6af82c7 327
5899fa0e
DW
328Gateway:: The IP address of the network's default gateway. On layer 3 zones
329 (Simple/EVPN plugins), it will be deployed on the VNet.
3093e62d 330
bcb72b25
SL
331SNAT:: Enable Source NAT which allows VMs from inside a
332 VNet to connect to the outside network by forwarding the packets to the nodes
333 outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
334 Optional.
335
336DNS Zone Prefix:: Add a prefix to the domain registration, like
337 <hostname>.prefix.<domain> Optional.
a6af82c7
AD
338
339
a6af82c7
AD
340[[pvesdn_config_controllers]]
341Controllers
342-----------
343
bcb72b25
SL
344Some zones implement a separated control and data plane that require an external
345external controller to manage the VNet's control plane.
346
347Currently, only the `EVPN` zone requires an external controller.
348
8ac25ffe 349
4e652aba 350[[pvesdn_controller_plugin_evpn]]
ee6e18c4
TL
351EVPN Controller
352~~~~~~~~~~~~~~~
1556b768 353
bcb72b25
SL
354The `EVPN`, zone requires an external controller to manage the control plane.
355The EVPN controller plugin configures the Free Range Routing (frr) router.
356
357To enable the EVPN controller, you need to install frr on every node that shall
358participate in the EVPN zone.
1556b768
AD
359
360----
a6af82c7 361apt install frr frr-pythontools
1556b768
AD
362----
363
bcb72b25 364EVPN controller configuration options:
1556b768 365
bcb72b25 366ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
5899fa0e
DW
367 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
368 breaking global routing by mistake.
1556b768 369
bcb72b25
SL
370Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
371 external nodes or route reflector servers)
1556b768 372
1556b768 373
a6af82c7
AD
374[[pvesdn_controller_plugin_BGP]]
375BGP Controller
bcb72b25 376~~~~~~~~~~~~~~
1556b768 377
5899fa0e
DW
378The BGP controller is not used directly by a zone.
379You can use it to configure FRR to manage BGP peers.
1556b768 380
5899fa0e 381For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
065b2147 382It can also be used to export EVPN routes to an external BGP peer.
4cf52a0c 383
065b2147
FG
384NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
385controller.
1556b768 386
bcb72b25 387BGP controller configuration options:
1556b768 388
bcb72b25 389Node:: The node of this BGP controller
9a4f3c95 390
bcb72b25 391ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
5899fa0e
DW
392 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
393 you could break global routing by mistake.
a6af82c7 394
bcb72b25 395Peer:: A list of peer IP addresses you want to communicate with using the
5899fa0e 396 underlying BGP network.
a6af82c7 397
bcb72b25 398EBGP:: If your peer's remote-AS is different, this enables EBGP.
a6af82c7 399
bcb72b25 400Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
5899fa0e 401 (for multipath).
a6af82c7 402
5899fa0e
DW
403ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
404 not directly connected or they use loopback.
a6af82c7 405
5899fa0e 406bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
f4e692a3 407
e5900013
AD
408
409[[pvesdn_controller_plugin_ISIS]]
410ISIS Controller
411~~~~~~~~~~~~~~~
412
413The ISIS controller is not used directly by a zone.
bcb72b25 414You can use it to configure FRR to export EVPN routes to an ISIS domain.
e5900013 415
bcb72b25 416ISIS controller configuration options:
e5900013 417
bcb72b25 418Node:: The node of this ISIS controller.
e5900013 419
bcb72b25 420Domain:: A unique ISIS domain.
e5900013 421
bcb72b25 422Network Entity Title:: A Unique ISIS network address that identifies this node.
e5900013 423
bcb72b25 424Interfaces:: A list of physical interface(s) used by ISIS.
e5900013 425
bcb72b25 426Loopback:: Use a loopback or dummy interface as the source of the EVPN network
e5900013
AD
427 (for multipath).
428
bcb72b25 429
a6af82c7 430[[pvesdn_config_ipam]]
bcb72b25
SL
431IPAM
432----
433
434IP Address Management (IPAM) tools manage the IP addresses of clients on the
435network. SDN in {pve} uses IPAM for example to find free IP addresses for new
436guests.
a6af82c7 437
bcb72b25 438A single IPAM instance can be associated with one or more zones.
a6af82c7
AD
439
440
441[[pvesdn_ipam_plugin_pveipam]]
bcb72b25
SL
442PVE IPAM Plugin
443~~~~~~~~~~~~~~~
a6af82c7 444
bcb72b25 445The default built-in IPAM for your {pve} cluster.
a6af82c7 446
a6af82c7 447
bcb72b25
SL
448[[pvesdn_ipam_plugin_netbox]]
449NetBox IPAM Plugin
450~~~~~~~~~~~~~
a6af82c7 451
bcb72b25
SL
452link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
453Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
a6af82c7 454
bcb72b25
SL
455To integrate NetBox with {pve} SDN, create an API token in NetBox as described
456here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
5899fa0e 457
bcb72b25 458The NetBox configuration properties are:
5899fa0e 459
bcb72b25 460URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
a6af82c7 461
bcb72b25 462Token:: An API access token
3093e62d 463
a6af82c7 464
bcb72b25
SL
465[[pvesdn_ipam_plugin_phpipam]]
466phpIPAM Plugin
467~~~~~~~~~~~~~~
a6af82c7 468
bcb72b25
SL
469In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
470an API token with admin privileges to the application.
471
472The phpIPAM configuration properties are:
a6af82c7 473
bcb72b25
SL
474URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
475
476Token:: An API access token
477
478Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
479 installations use `sectionid=1` for customers.
5899fa0e 480
a6af82c7
AD
481
482[[pvesdn_config_dns]]
3093e62d 483DNS
a6af82c7 484---
3093e62d
TL
485
486The DNS plugin in {pve} SDN is used to define a DNS API server for registration
5899fa0e
DW
487of your hostname and IP address. A DNS configuration is associated with one or
488more zones, to provide DNS registration for all the subnet IPs configured for
3093e62d 489a zone.
a6af82c7
AD
490
491[[pvesdn_dns_plugin_powerdns]]
5899fa0e 492PowerDNS Plugin
a6af82c7
AD
493~~~~~~~~~~~~~~~
494https://doc.powerdns.com/authoritative/http-api/index.html
495
5899fa0e 496You need to enable the web server and the API in your PowerDNS config:
1556b768 497
40f4dfc2 498----
a6af82c7
AD
499api=yes
500api-key=arandomgeneratedstring
501webserver=yes
502webserver-port=8081
40f4dfc2 503----
40f4dfc2 504
5899fa0e 505The PowerDNS configuration options are:
a6af82c7 506
3093e62d 507url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
5899fa0e 508
3093e62d 509key:: An API access key
5899fa0e 510
3093e62d 511ttl:: The default TTL for records
1556b768 512
1556b768 513
bcb72b25 514[[pvesdn_setup_examples]]
a6af82c7
AD
515Examples
516--------
517
bcb72b25
SL
518This section presents multiple configuration examples tailored for common SDN
519use cases. It aims to offer tangible implementations, providing additional
520details to enhance comprehension of the available configuration options.
1556b768 521
ee6e18c4 522
bcb72b25
SL
523[[pvesdn_setup_example_simple]]
524Simple Zone Example
525~~~~~~~~~~~~~~~~~~~
1556b768 526
bcb72b25
SL
527Simple zone networks create an isolated network for quests on a single host to
528connect to each other.
1556b768 529
bcb72b25
SL
530TIP: connection between quests are possible if all quests reside on a same host
531but cannot be reached on other nodes.
1556b768 532
bcb72b25
SL
533* Create a simple zone named `simple`.
534* Add a VNet names `vnet1`.
535* Create a Subnet with a gateway and the SNAT option enabled.
536* This creates a network bridge `vnet1` on the node. Assign this bridge to the
537 quests that shall join the network and configure an IP address.
1556b768 538
bcb72b25
SL
539The network interface configuration in two VMs may look like this which allows
540them to communicate via the 10.0.1.0/24 network.
1556b768
AD
541
542----
bcb72b25
SL
543allow-hotplug ens19
544iface ens19 inet static
545 address 10.0.1.14/24
546----
1556b768 547
bcb72b25
SL
548----
549allow-hotplug ens19
550iface ens19 inet static
551 address 10.0.1.15/24
552----
1556b768 553
bcb72b25
SL
554
555[[pvesdn_setup_example_nat]]
556Source NAT Example
557~~~~~~~~~~~~~~~~~~
558
559If you want to allow outgoing connections for quests in the simple network zone
560the simple zone offers a Source NAT (SNAT) option.
561
562Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
563Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
564
565----
566Subnet: 172.16.0.0/24
567Gateway: 172.16.0.1
568SNAT: checked
1556b768
AD
569----
570
bcb72b25
SL
571In the quests configure the static IP address inside the subnet's IP range.
572
573The node itself will join this network with the Gateway IP '172.16.0.1' and
574function as the NAT gateway for quests within the subnet range.
575
576
577[[pvesdn_setup_example_vlan]]
578VLAN Setup Example
579~~~~~~~~~~~~~~~~~~
580
581When VMs on different nodes need to communicate through an isolated network, the
582VLAN zone allows network level isolation using VLAN tags.
583
584Create a VLAN zone named `myvlanzone`:
1556b768
AD
585
586----
bcb72b25
SL
587ID: myvlanzone
588Bridge: vmbr0
1556b768
AD
589----
590
bcb72b25
SL
591Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
592`myvlanzone`.
1556b768
AD
593
594----
bcb72b25
SL
595ID: myvnet1
596Zone: myvlanzone
597Tag: 10
1556b768
AD
598----
599
ee6e18c4 600Apply the configuration through the main SDN panel, to create VNets locally on
5899fa0e 601each node.
1556b768 602
bcb72b25 603Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
1556b768 604
ee6e18c4 605Use the following network configuration for this VM:
1556b768
AD
606
607----
608auto eth0
609iface eth0 inet static
ee6e18c4 610 address 10.0.3.100/24
1556b768
AD
611----
612
bcb72b25
SL
613Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
614`myvnet1` as vm1.
ee6e18c4
TL
615
616Use the following network configuration for this VM:
617
1556b768
AD
618----
619auto eth0
620iface eth0 inet static
ee6e18c4 621 address 10.0.3.101/24
1556b768
AD
622----
623
bcb72b25 624Following this, you should be able to ping between both VMs using that network.
1556b768
AD
625
626
4e652aba
TL
627[[pvesdn_setup_example_qinq]]
628QinQ Setup Example
a6af82c7 629~~~~~~~~~~~~~~~~~~
ee6e18c4 630
1556b768 631
bcb72b25
SL
632This example configures two QinQ zones and adds two VMs to each zone to
633demonstrate the additional layer of VLAN tags which allows the configuration of
634more isolated VLANs.
1556b768 635
bcb72b25
SL
636A typical use case for this configuration is a hosting provider that provides an
637isolated network to customers for VM communication but isolates the VMs from
638other customers.
1556b768 639
bcb72b25 640Create a QinQ zone named `qinqzone1` with service VLAN 20
1556b768
AD
641
642----
bcb72b25
SL
643ID: qinqzone1
644Bridge: vmbr0
645Service VLAN: 20
1556b768
AD
646----
647
bcb72b25 648Create another QinQ zone named `qinqzone2` with service VLAN 30
1556b768 649----
bcb72b25
SL
650ID: qinqzone2
651Bridge: vmbr0
652Service VLAN: 30
1556b768
AD
653----
654
bcb72b25
SL
655Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
656`qinqzone1` zone.
1556b768
AD
657
658----
bcb72b25
SL
659ID: qinqvnet1
660Zone: qinqzone1
661Tag: 100
1556b768
AD
662----
663
bcb72b25 664Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
1556b768
AD
665
666----
bcb72b25
SL
667ID: qinqvnet2
668Zone: qinqzone2
669Tag: 100
1556b768
AD
670----
671
ee6e18c4 672Apply the configuration on the main SDN web-interface panel to create VNets
bcb72b25 673locally on each node.
1556b768 674
bcb72b25
SL
675Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
676interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
677`qinqvnet2`.
ee6e18c4 678
bcb72b25
SL
679Inside the VM, configure the IP addresses of the interfaces, for example via
680`/etc/network/interfaces`:
ee6e18c4 681
1556b768
AD
682----
683auto eth0
684iface eth0 inet static
bcb72b25 685 address 10.0.3.101/24
1556b768 686----
bcb72b25
SL
687// TODO: systemd-network example
688Configure all four VMs to have IP addresses from the '10.0.3.101' to
689'10.0.3.104' range.
1556b768 690
bcb72b25 691Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
5899fa0e 692between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
bcb72b25 693'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
1556b768 694
1556b768 695
4e652aba 696[[pvesdn_setup_example_vxlan]]
ee6e18c4 697VXLAN Setup Example
a6af82c7 698~~~~~~~~~~~~~~~~~~~
ee6e18c4 699
bcb72b25
SL
700The example assumes a cluster with three nodes, with the node IP addresses
701192.168.0.1, 192.168.0.2 and 192.168.0.3.
1556b768 702
bcb72b25
SL
703Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
704peer address list. Use the default MTU of 1450 or configure accordingly.
1556b768 705
1556b768 706----
bcb72b25
SL
707ID: myvxlanzone
708Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
1556b768
AD
709----
710
bcb72b25 711Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
ee6e18c4 712previously.
1556b768
AD
713
714----
bcb72b25
SL
715ID: vxvnet1
716Zone: myvxlanzone
717Tag: 100000
1556b768
AD
718----
719
ee6e18c4
TL
720Apply the configuration on the main SDN web-interface panel to create VNets
721locally on each nodes.
1556b768 722
bcb72b25 723Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
1556b768 724
5899fa0e 725Use the following network configuration for this VM (note the lower MTU).
1556b768
AD
726
727----
728auto eth0
729iface eth0 inet static
bcb72b25
SL
730 address 10.0.3.100/24
731 mtu 1450
1556b768
AD
732----
733
bcb72b25
SL
734Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
735`vxvnet1` as vm1.
ee6e18c4
TL
736
737Use the following network configuration for this VM:
738
1556b768
AD
739----
740auto eth0
741iface eth0 inet static
bcb72b25
SL
742 address 10.0.3.101/24
743 mtu 1450
1556b768
AD
744----
745
ee6e18c4 746Then, you should be able to ping between between 'vm1' and 'vm2'.
1556b768
AD
747
748
4e652aba
TL
749[[pvesdn_setup_example_evpn]]
750EVPN Setup Example
a6af82c7 751~~~~~~~~~~~~~~~~~~
ee6e18c4 752
bcb72b25
SL
753The example assumes a cluster with three nodes (node1, node2, node3) with IP
754addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
1556b768 755
5899fa0e
DW
756Create an EVPN controller, using a private ASN number and the above node
757addresses as peers.
1556b768
AD
758
759----
bcb72b25
SL
760ID: myevpnctl
761ASN#: 65000
762Peers: 192.168.0.1,192.168.0.2,192.168.0.3
1556b768
AD
763----
764
bcb72b25
SL
765Create an EVPN zone named `myevpnzone`, assign the previously created
766EVPN-controller and define 'node1' and 'node2' as exit nodes.
a6af82c7 767
1556b768 768----
bcb72b25
SL
769ID: myevpnzone
770VRF VXLAN Tag: 10000
771Controller: myevpnctl
772MTU: 1450
773VNet MAC Address: 32:F4:05:FE:6C:0A
774Exit Nodes: node1,node2
1556b768
AD
775----
776
bcb72b25
SL
777Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
778
1556b768 779----
bcb72b25
SL
780ID: myvnet1
781Zone: myevpnzone
782Tag: 11000
1556b768
AD
783----
784
bcb72b25 785Create a subnet on `myvnet1`:
9f819242 786
a6af82c7 787----
bcb72b25
SL
788Subnet: 10.0.1.0/24
789Gateway: 10.0.1.1
a6af82c7
AD
790----
791
bcb72b25 792Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
1556b768
AD
793
794----
bcb72b25
SL
795ID: myvnet2
796Zone: myevpnzone
797Tag: 12000
1556b768
AD
798----
799
bcb72b25 800Create a different subnet on `myvnet2``:
9f819242 801
a6af82c7 802----
bcb72b25
SL
803Subnet: 10.0.2.0/24
804Gateway: 10.0.2.1
a6af82c7
AD
805----
806
5899fa0e 807Apply the configuration from the main SDN web-interface panel to create VNets
bcb72b25 808locally on each node and generate the FRR configuration.
1556b768 809
bcb72b25 810Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
1556b768 811
bcb72b25 812Use the following network configuration for 'vm1':
1556b768
AD
813
814----
815auto eth0
816iface eth0 inet static
bcb72b25
SL
817 address 10.0.1.100/24
818 gateway 10.0.1.1
819 mtu 1450
1556b768
AD
820----
821
bcb72b25
SL
822Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
823`myvnet2`.
ee6e18c4 824
bcb72b25 825Use the following network configuration for 'vm2':
ee6e18c4 826
1556b768
AD
827----
828auto eth0
829iface eth0 inet static
bcb72b25
SL
830 address 10.0.2.100/24
831 gateway 10.0.2.1
832 mtu 1450
1556b768
AD
833----
834
835
bcb72b25 836Now you should be able to ping vm2 from vm1, and vm1 from vm2.
1556b768 837
bcb72b25 838If you ping an external IP from 'vm2' on the non-gateway node3, the packet
a6af82c7 839will go to the configured 'myvnet2' gateway, then will be routed to the exit
ee6e18c4
TL
840nodes ('node1' or 'node2') and from there it will leave those nodes over the
841default gateway configured on node1 or node2.
1556b768 842
5899fa0e
DW
843NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
844networks to node1 and node2 on your external gateway, so that the public network
845can reply back.
1556b768 846
ee6e18c4
TL
847If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
848and 10.0.2.0/24 in this example), will be announced dynamically.
ce84ae7e
AD
849
850
bcb72b25 851[[pvesdn_notes]]
ce84ae7e
AD
852Notes
853-----
854
4368fce0
AD
855Multiple EVPN Exit Nodes
856~~~~~~~~~~~~~~~~~~~~~~~~
857
5b105f3e
TL
858If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
859Reverse Path Filter) option, because packets can arrive at one node but go out
860from another node.
4368fce0 861
bcb72b25
SL
862Add the following to `/etc/sysctl.conf`:
863
4368fce0
AD
864-----
865net.ipv4.conf.default.rp_filter=0
866net.ipv4.conf.all.rp_filter=0
867-----
868
448c1d39
TL
869VXLAN IPSEC Encryption
870~~~~~~~~~~~~~~~~~~~~~~
5899fa0e 871
bcb72b25
SL
872To add IPSEC encryption on top of a VXLAN, this example shows how to use
873`strongswan`.
874
875You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
876IPv6 to handle encryption.
ce84ae7e 877
448c1d39
TL
878So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
879+ 50 (VXLAN) == 1500).
ce84ae7e 880
bcb72b25
SL
881Install strongswan on the host.
882
ce84ae7e 883----
8bfa192d 884apt install strongswan
ce84ae7e
AD
885----
886
bcb72b25 887Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
448c1d39 888the VXLAN UDP port '4789'.
ce84ae7e
AD
889
890----
891conn %default
448c1d39 892 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
ce84ae7e 893 esp=aes256-sha1!
448c1d39 894 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
ce84ae7e
AD
895
896conn output
897 rightsubnet=%dynamic[udp/4789]
898 right=%any
899 type=transport
900 authby=psk
901 auto=route
902
903conn input
904 leftsubnet=%dynamic[udp/4789]
905 type=transport
906 authby=psk
907 auto=route
908----
909
bcb72b25 910Generate a pre-shared key with:
ce84ae7e
AD
911
912----
913openssl rand -base64 128
914----
915
bcb72b25 916and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
ce84ae7e
AD
917
918----
919: PSK <generatedbase64key>
920----
448c1d39 921
bcb72b25 922Copy the PSK and the configuration to all nodes participating in the VXLAN network.