]> git.proxmox.com Git - pve-docs.git/blob - pvesdn.adoc
sdn: overall language and consistency rework
[pve-docs.git] / pvesdn.adoc
1 [[chapter_pvesdn]]
2 Software-Defined Network
3 ========================
4 ifndef::manvolnum[]
5 :pve-toplevel:
6 endif::manvolnum[]
7
8 The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create
9 virtual networks (VNets) at the datacenter level.
10
11 WARNING: SDN is currently an **experimental feature** in {pve}. This
12 documentation for it is also still under development. Ask on our
13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
14
15
16 [[pvesdn_installation]]
17 Installation
18 ------------
19
20 To enable the experimental Software-Defined Network (SDN) integration, you need
21 to install the `libpve-network-perl` package on every node:
22
23 ----
24 apt update
25 apt install libpve-network-perl
26 ----
27
28 NOTE: {pve} version 7 and above have the `ifupdown2` package installed by
29 default. If you originally installed your system with an older version, you need
30 to explicitly install the `ifupdown2` package.
31
32 After installation, you need to add the following line to the end of the
33 `/etc/network/interfaces` configuration file, so that the SDN configuration gets
34 included and activated.
35
36 ----
37 source /etc/network/interfaces.d/*
38 ----
39
40
41 [[pvesdn_overview]]
42 Overview
43 --------
44
45 The {pve} SDN allows for separation and fine-grained control of virtual guest
46 networks, using flexible, software-controlled configurations.
47
48 Separation is managed through *zones*, virtual networks (*VNets*), and
49 *subnets*. A zone is its own virtually separated network area. A VNet is a
50 virtual network that belongs to a zone. A subnet is an IP range inside a VNet.
51
52 Depending on the type of the zone, the network behaves differently and offers
53 specific features, advantages, and limitations.
54
55 Use cases for SDN range from an isolated private network on each individual node
56 to complex overlay networks across multiple PVE clusters on different locations.
57
58 After configuring an VNet in the cluster-wide datacenter SDN administration
59 interface, it is available as a common Linux bridge, locally on each node, to be
60 assigned to VMs and Containers.
61
62
63 [[pvesdn_main_configuration]]
64 Main Configuration
65 ~~~~~~~~~~~~~~~~~~
66
67 Configuration is done at the web UI at datacenter level and is saved in files
68 located in the shared configuration file system at `/etc/pve/sdn`.
69
70 On the web interface, SDN features the following sections:
71
72 * SDN:: Here you get an overview of the current active SDN state, and you can
73 apply all pending changes to the whole cluster.
74
75 * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated
76 network zones
77
78 * xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and
79 manage subnets
80
81 The Options category allows adding and managing additional services to be used
82 in your SDN setup.
83
84 * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing
85 in complex setups
86
87 * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for
88 guests
89
90 * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering
91 virtual guests' hostname and IP
92 addresses
93
94 [[pvesdn_config_main_sdn]]
95 SDN
96 ~~~
97
98 This is the main status panel. Here you can see the deployment status of zones
99 on different nodes.
100
101 Pressing the 'Apply' button reloads the local configuration on all cluster
102 nodes.
103
104 [[pvesdn_config_zone]]
105 Zones
106 -----
107
108 A zone defines a virtually separated network. Zones are restricted to
109 specific nodes and assigned permissions, in order to restrict users to a certain
110 zone and its contained VNets.
111
112 Different technologies can be used for separation:
113
114 * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
115
116 * VLAN: Virtual LANs are the classic method of subdividing a LAN
117
118 * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`)
119
120 * VXLAN: Layer 2 VXLAN network via a UDP tunnel
121
122 * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
123
124
125 [[pvesdn_config_common_options]]
126 Common Options
127 ~~~~~~~~~~~~~~
128
129 The following options are available for all zone types:
130
131 Nodes:: The nodes which the zone and associated VNets should be deployed on.
132
133 IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the
134 zone. Optional, defaults to `pve`.
135
136 DNS:: DNS API server. Optional.
137
138 ReverseDNS:: Reverse DNS API server. Optional.
139
140 DNSZone:: DNS domain name. Used to register hostnames, such as
141 `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional.
142
143
144 [[pvesdn_zone_plugin_simple]]
145 Simple Zones
146 ~~~~~~~~~~~~
147
148 This is the simplest plugin. It will create an isolated VNet bridge. This
149 bridge is not linked to a physical interface, and VM traffic is only local on
150 each the node.
151 It can be used in NAT or routed setups.
152
153
154 [[pvesdn_zone_plugin_vlan]]
155 VLAN Zones
156 ~~~~~~~~~~
157
158 The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
159 node's physical interface. It uses VLAN tagging defined in the VNet to isolate
160 the network segments. This allows connectivity of VMs between different nodes.
161
162 VLAN zone configuration options:
163
164 Bridge:: The local bridge or OVS switch, already configured on *each* node that
165 allows node-to-node connection.
166
167
168 [[pvesdn_zone_plugin_qinq]]
169 QinQ Zones
170 ~~~~~~~~~~
171
172 QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
173 isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN')
174 whereas the inner VLAN tag is defined by the VNet.
175
176 NOTE: Your physical network switches must support stacked VLANs for this
177 configuration.
178
179 QinQ zone configuration options:
180
181 Bridge:: A local, VLAN-aware bridge that is already configured on each local
182 node
183
184 Service VLAN:: The main VLAN tag of this zone
185
186 Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or
187 802.1ad service VLAN type.
188
189 MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
190 For example, you must reduce the MTU to `1496` if you physical interface MTU is
191 `1500`.
192
193
194 [[pvesdn_zone_plugin_vxlan]]
195 VXLAN Zones
196 ~~~~~~~~~~~
197
198 The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
199 (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP
200 datagrams using the default destination port `4789`.
201
202 You have to configure the underlay network yourself to enable UDP connectivity
203 between all peers.
204
205 You can, for example, create a VXLAN overlay network on top of public internet,
206 appearing to the VMs as if they share the same local Layer 2 network.
207
208 WARNING: VXLAN on its own does does not provide any encryption. When joining
209 multiple sites via VXLAN, make sure to establish a secure connection between
210 the site, for example by using a site-to-site VPN.
211
212 VXLAN zone configuration options:
213
214 Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This
215 can be external nodes reachable at this IP address.
216 All nodes in the cluster need to be mentioned here.
217
218 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
219 lower than the outgoing physical interface.
220
221
222 [[pvesdn_zone_plugin_evpn]]
223 EVPN Zones
224 ~~~~~~~~~~
225
226 The EVPN zone creates a routable Layer 3 network, capable of spanning across
227 multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
228 the routing protocol.
229
230 The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
231 IP is the same on each node, meaning a virtual guest can use this address as
232 gateway.
233
234 Routing can work across VNets from different zones through a VRF (Virtual
235 Routing and Forwarding) interface.
236
237 EVPN zone configuration options:
238
239 VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets.
240 It must be different than the VXLAN-ID of the VNets.
241
242 Controller:: The EVPN-controller to use for this zone. (See controller plugins
243 section).
244
245 VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this
246 zone. Will be auto-generated if not defined.
247
248 Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN
249 network, through the real network. The configured nodes will announce a
250 default route in the EVPN network. Optional.
251
252 Primary Exit Node:: If you use multiple exit nodes, force traffic through this
253 primary exit node, instead of load-balancing on all nodes. Optional but
254 necessary if you want to use SNAT or if your upstream router doesn't support
255 ECMP.
256
257 Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT
258 service from an exit node. (By default, the exit nodes only allow forwarding
259 traffic between real network and EVPN network). Optional.
260
261 Advertise Subnets:: Announce the full subnet in the EVPN network.
262 If you have silent VMs/CTs (for example, if you have multiple IPs and the
263 anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be
264 able to be reached inside the EVPN network). Optional.
265
266 Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery)
267 packets. This is required if you use floating IPs in your VMs (IP and MAC
268 addresses are being moved between systems). Optional.
269
270 Route-target Import:: Allows you to import a list of external EVPN route
271 targets. Used for cross-DC or different EVPN network interconnects. Optional.
272
273 MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
274 less than the maximal MTU of the outgoing physical interface. Optional,
275 defaults to 1450.
276
277
278 [[pvesdn_config_vnets]]
279 VNets
280 -----
281
282 After creating a virtual network (VNet) through the SDN GUI, a local network
283 interface with the same name is available on each node. To connect a guest to the
284 VNet, assign the interface to the guest and set the IP address accordingly.
285
286 Depending on the zone, these options have different meanings and are explained
287 in the respective zone section in this document.
288
289 WARNING: In the current state, some options may have no effect or won't work in
290 certain zones.
291
292 VNet configuration options:
293
294 ID:: An up to 8 character ID to identify a VNet
295
296 Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional
297
298 Zone:: The associated zone for this VNet
299
300 Tag:: The unique VLAN or VXLAN ID
301
302 VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration
303 in the quest.
304
305
306 [[pvesdn_config_subnet]]
307 Subnets
308 -------
309
310 A subnet define a specific IP range, described by the CIDR network address.
311 Each VNet, can have one or more subnets.
312
313 A subnet can be used to:
314
315 * Restrict the IP addresses you can define on a specific VNet
316 * Assign routes/gateways on a VNet in layer 3 zones
317 * Enable SNAT on a VNet in layer 3 zones
318 * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
319 * DNS registration through DNS plugins
320
321 If an IPAM server is associated with the subnet zone, the subnet prefix will be
322 automatically registered in the IPAM.
323
324 Subnet configuration options:
325
326 ID:: A CIDR network address, for example 10.0.0.0/8
327
328 Gateway:: The IP address of the network's default gateway. On layer 3 zones
329 (Simple/EVPN plugins), it will be deployed on the VNet.
330
331 SNAT:: Enable Source NAT which allows VMs from inside a
332 VNet to connect to the outside network by forwarding the packets to the nodes
333 outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
334 Optional.
335
336 DNS Zone Prefix:: Add a prefix to the domain registration, like
337 <hostname>.prefix.<domain> Optional.
338
339
340 [[pvesdn_config_controllers]]
341 Controllers
342 -----------
343
344 Some zones implement a separated control and data plane that require an external
345 external controller to manage the VNet's control plane.
346
347 Currently, only the `EVPN` zone requires an external controller.
348
349
350 [[pvesdn_controller_plugin_evpn]]
351 EVPN Controller
352 ~~~~~~~~~~~~~~~
353
354 The `EVPN`, zone requires an external controller to manage the control plane.
355 The EVPN controller plugin configures the Free Range Routing (frr) router.
356
357 To enable the EVPN controller, you need to install frr on every node that shall
358 participate in the EVPN zone.
359
360 ----
361 apt install frr frr-pythontools
362 ----
363
364 EVPN controller configuration options:
365
366 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
367 number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
368 breaking global routing by mistake.
369
370 Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be
371 external nodes or route reflector servers)
372
373
374 [[pvesdn_controller_plugin_BGP]]
375 BGP Controller
376 ~~~~~~~~~~~~~~
377
378 The BGP controller is not used directly by a zone.
379 You can use it to configure FRR to manage BGP peers.
380
381 For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
382 It can also be used to export EVPN routes to an external BGP peer.
383
384 NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP
385 controller.
386
387 BGP controller configuration options:
388
389 Node:: The node of this BGP controller
390
391 ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN
392 number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
393 you could break global routing by mistake.
394
395 Peer:: A list of peer IP addresses you want to communicate with using the
396 underlying BGP network.
397
398 EBGP:: If your peer's remote-AS is different, this enables EBGP.
399
400 Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network
401 (for multipath).
402
403 ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are
404 not directly connected or they use loopback.
405
406 bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN.
407
408
409 [[pvesdn_controller_plugin_ISIS]]
410 ISIS Controller
411 ~~~~~~~~~~~~~~~
412
413 The ISIS controller is not used directly by a zone.
414 You can use it to configure FRR to export EVPN routes to an ISIS domain.
415
416 ISIS controller configuration options:
417
418 Node:: The node of this ISIS controller.
419
420 Domain:: A unique ISIS domain.
421
422 Network Entity Title:: A Unique ISIS network address that identifies this node.
423
424 Interfaces:: A list of physical interface(s) used by ISIS.
425
426 Loopback:: Use a loopback or dummy interface as the source of the EVPN network
427 (for multipath).
428
429
430 [[pvesdn_config_ipam]]
431 IPAM
432 ----
433
434 IP Address Management (IPAM) tools manage the IP addresses of clients on the
435 network. SDN in {pve} uses IPAM for example to find free IP addresses for new
436 guests.
437
438 A single IPAM instance can be associated with one or more zones.
439
440
441 [[pvesdn_ipam_plugin_pveipam]]
442 PVE IPAM Plugin
443 ~~~~~~~~~~~~~~~
444
445 The default built-in IPAM for your {pve} cluster.
446
447
448 [[pvesdn_ipam_plugin_netbox]]
449 NetBox IPAM Plugin
450 ~~~~~~~~~~~~~
451
452 link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP
453 Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.
454
455 To integrate NetBox with {pve} SDN, create an API token in NetBox as described
456 here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens
457
458 The NetBox configuration properties are:
459
460 URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api`
461
462 Token:: An API access token
463
464
465 [[pvesdn_ipam_plugin_phpipam]]
466 phpIPAM Plugin
467 ~~~~~~~~~~~~~~
468
469 In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add
470 an API token with admin privileges to the application.
471
472 The phpIPAM configuration properties are:
473
474 URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
475
476 Token:: An API access token
477
478 Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default
479 installations use `sectionid=1` for customers.
480
481
482 [[pvesdn_config_dns]]
483 DNS
484 ---
485
486 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
487 of your hostname and IP address. A DNS configuration is associated with one or
488 more zones, to provide DNS registration for all the subnet IPs configured for
489 a zone.
490
491 [[pvesdn_dns_plugin_powerdns]]
492 PowerDNS Plugin
493 ~~~~~~~~~~~~~~~
494 https://doc.powerdns.com/authoritative/http-api/index.html
495
496 You need to enable the web server and the API in your PowerDNS config:
497
498 ----
499 api=yes
500 api-key=arandomgeneratedstring
501 webserver=yes
502 webserver-port=8081
503 ----
504
505 The PowerDNS configuration options are:
506
507 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
508
509 key:: An API access key
510
511 ttl:: The default TTL for records
512
513
514 [[pvesdn_setup_examples]]
515 Examples
516 --------
517
518 This section presents multiple configuration examples tailored for common SDN
519 use cases. It aims to offer tangible implementations, providing additional
520 details to enhance comprehension of the available configuration options.
521
522
523 [[pvesdn_setup_example_simple]]
524 Simple Zone Example
525 ~~~~~~~~~~~~~~~~~~~
526
527 Simple zone networks create an isolated network for quests on a single host to
528 connect to each other.
529
530 TIP: connection between quests are possible if all quests reside on a same host
531 but cannot be reached on other nodes.
532
533 * Create a simple zone named `simple`.
534 * Add a VNet names `vnet1`.
535 * Create a Subnet with a gateway and the SNAT option enabled.
536 * This creates a network bridge `vnet1` on the node. Assign this bridge to the
537 quests that shall join the network and configure an IP address.
538
539 The network interface configuration in two VMs may look like this which allows
540 them to communicate via the 10.0.1.0/24 network.
541
542 ----
543 allow-hotplug ens19
544 iface ens19 inet static
545 address 10.0.1.14/24
546 ----
547
548 ----
549 allow-hotplug ens19
550 iface ens19 inet static
551 address 10.0.1.15/24
552 ----
553
554
555 [[pvesdn_setup_example_nat]]
556 Source NAT Example
557 ~~~~~~~~~~~~~~~~~~
558
559 If you want to allow outgoing connections for quests in the simple network zone
560 the simple zone offers a Source NAT (SNAT) option.
561
562 Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a
563 Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option.
564
565 ----
566 Subnet: 172.16.0.0/24
567 Gateway: 172.16.0.1
568 SNAT: checked
569 ----
570
571 In the quests configure the static IP address inside the subnet's IP range.
572
573 The node itself will join this network with the Gateway IP '172.16.0.1' and
574 function as the NAT gateway for quests within the subnet range.
575
576
577 [[pvesdn_setup_example_vlan]]
578 VLAN Setup Example
579 ~~~~~~~~~~~~~~~~~~
580
581 When VMs on different nodes need to communicate through an isolated network, the
582 VLAN zone allows network level isolation using VLAN tags.
583
584 Create a VLAN zone named `myvlanzone`:
585
586 ----
587 ID: myvlanzone
588 Bridge: vmbr0
589 ----
590
591 Create a VNet named `myvnet1` with VLAN tag 10 and the previously created
592 `myvlanzone`.
593
594 ----
595 ID: myvnet1
596 Zone: myvlanzone
597 Tag: 10
598 ----
599
600 Apply the configuration through the main SDN panel, to create VNets locally on
601 each node.
602
603 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
604
605 Use the following network configuration for this VM:
606
607 ----
608 auto eth0
609 iface eth0 inet static
610 address 10.0.3.100/24
611 ----
612
613 Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet
614 `myvnet1` as vm1.
615
616 Use the following network configuration for this VM:
617
618 ----
619 auto eth0
620 iface eth0 inet static
621 address 10.0.3.101/24
622 ----
623
624 Following this, you should be able to ping between both VMs using that network.
625
626
627 [[pvesdn_setup_example_qinq]]
628 QinQ Setup Example
629 ~~~~~~~~~~~~~~~~~~
630
631
632 This example configures two QinQ zones and adds two VMs to each zone to
633 demonstrate the additional layer of VLAN tags which allows the configuration of
634 more isolated VLANs.
635
636 A typical use case for this configuration is a hosting provider that provides an
637 isolated network to customers for VM communication but isolates the VMs from
638 other customers.
639
640 Create a QinQ zone named `qinqzone1` with service VLAN 20
641
642 ----
643 ID: qinqzone1
644 Bridge: vmbr0
645 Service VLAN: 20
646 ----
647
648 Create another QinQ zone named `qinqzone2` with service VLAN 30
649 ----
650 ID: qinqzone2
651 Bridge: vmbr0
652 Service VLAN: 30
653 ----
654
655 Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created
656 `qinqzone1` zone.
657
658 ----
659 ID: qinqvnet1
660 Zone: qinqzone1
661 Tag: 100
662 ----
663
664 Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone.
665
666 ----
667 ID: qinqvnet2
668 Zone: qinqzone2
669 Tag: 100
670 ----
671
672 Apply the configuration on the main SDN web-interface panel to create VNets
673 locally on each node.
674
675 Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
676 interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge
677 `qinqvnet2`.
678
679 Inside the VM, configure the IP addresses of the interfaces, for example via
680 `/etc/network/interfaces`:
681
682 ----
683 auto eth0
684 iface eth0 inet static
685 address 10.0.3.101/24
686 ----
687 // TODO: systemd-network example
688 Configure all four VMs to have IP addresses from the '10.0.3.101' to
689 '10.0.3.104' range.
690
691 Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as
692 between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs
693 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN.
694
695
696 [[pvesdn_setup_example_vxlan]]
697 VXLAN Setup Example
698 ~~~~~~~~~~~~~~~~~~~
699
700 The example assumes a cluster with three nodes, with the node IP addresses
701 192.168.0.1, 192.168.0.2 and 192.168.0.3.
702
703 Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the
704 peer address list. Use the default MTU of 1450 or configure accordingly.
705
706 ----
707 ID: myvxlanzone
708 Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3
709 ----
710
711 Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created
712 previously.
713
714 ----
715 ID: vxvnet1
716 Zone: myvxlanzone
717 Tag: 100000
718 ----
719
720 Apply the configuration on the main SDN web-interface panel to create VNets
721 locally on each nodes.
722
723 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`.
724
725 Use the following network configuration for this VM (note the lower MTU).
726
727 ----
728 auto eth0
729 iface eth0 inet static
730 address 10.0.3.100/24
731 mtu 1450
732 ----
733
734 Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet
735 `vxvnet1` as vm1.
736
737 Use the following network configuration for this VM:
738
739 ----
740 auto eth0
741 iface eth0 inet static
742 address 10.0.3.101/24
743 mtu 1450
744 ----
745
746 Then, you should be able to ping between between 'vm1' and 'vm2'.
747
748
749 [[pvesdn_setup_example_evpn]]
750 EVPN Setup Example
751 ~~~~~~~~~~~~~~~~~~
752
753 The example assumes a cluster with three nodes (node1, node2, node3) with IP
754 addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.
755
756 Create an EVPN controller, using a private ASN number and the above node
757 addresses as peers.
758
759 ----
760 ID: myevpnctl
761 ASN#: 65000
762 Peers: 192.168.0.1,192.168.0.2,192.168.0.3
763 ----
764
765 Create an EVPN zone named `myevpnzone`, assign the previously created
766 EVPN-controller and define 'node1' and 'node2' as exit nodes.
767
768 ----
769 ID: myevpnzone
770 VRF VXLAN Tag: 10000
771 Controller: myevpnctl
772 MTU: 1450
773 VNet MAC Address: 32:F4:05:FE:6C:0A
774 Exit Nodes: node1,node2
775 ----
776
777 Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`.
778
779 ----
780 ID: myvnet1
781 Zone: myevpnzone
782 Tag: 11000
783 ----
784
785 Create a subnet on `myvnet1`:
786
787 ----
788 Subnet: 10.0.1.0/24
789 Gateway: 10.0.1.1
790 ----
791
792 Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`.
793
794 ----
795 ID: myvnet2
796 Zone: myevpnzone
797 Tag: 12000
798 ----
799
800 Create a different subnet on `myvnet2``:
801
802 ----
803 Subnet: 10.0.2.0/24
804 Gateway: 10.0.2.1
805 ----
806
807 Apply the configuration from the main SDN web-interface panel to create VNets
808 locally on each node and generate the FRR configuration.
809
810 Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`.
811
812 Use the following network configuration for 'vm1':
813
814 ----
815 auto eth0
816 iface eth0 inet static
817 address 10.0.1.100/24
818 gateway 10.0.1.1
819 mtu 1450
820 ----
821
822 Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet
823 `myvnet2`.
824
825 Use the following network configuration for 'vm2':
826
827 ----
828 auto eth0
829 iface eth0 inet static
830 address 10.0.2.100/24
831 gateway 10.0.2.1
832 mtu 1450
833 ----
834
835
836 Now you should be able to ping vm2 from vm1, and vm1 from vm2.
837
838 If you ping an external IP from 'vm2' on the non-gateway node3, the packet
839 will go to the configured 'myvnet2' gateway, then will be routed to the exit
840 nodes ('node1' or 'node2') and from there it will leave those nodes over the
841 default gateway configured on node1 or node2.
842
843 NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24'
844 networks to node1 and node2 on your external gateway, so that the public network
845 can reply back.
846
847 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
848 and 10.0.2.0/24 in this example), will be announced dynamically.
849
850
851 [[pvesdn_notes]]
852 Notes
853 -----
854
855 Multiple EVPN Exit Nodes
856 ~~~~~~~~~~~~~~~~~~~~~~~~
857
858 If you have multiple gateway nodes, you should disable the `rp_filter` (Strict
859 Reverse Path Filter) option, because packets can arrive at one node but go out
860 from another node.
861
862 Add the following to `/etc/sysctl.conf`:
863
864 -----
865 net.ipv4.conf.default.rp_filter=0
866 net.ipv4.conf.all.rp_filter=0
867 -----
868
869 VXLAN IPSEC Encryption
870 ~~~~~~~~~~~~~~~~~~~~~~
871
872 To add IPSEC encryption on top of a VXLAN, this example shows how to use
873 `strongswan`.
874
875 You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for
876 IPv6 to handle encryption.
877
878 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
879 + 50 (VXLAN) == 1500).
880
881 Install strongswan on the host.
882
883 ----
884 apt install strongswan
885 ----
886
887 Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from
888 the VXLAN UDP port '4789'.
889
890 ----
891 conn %default
892 ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW
893 esp=aes256-sha1!
894 leftfirewall=yes # this is necessary when using Proxmox VE firewall rules
895
896 conn output
897 rightsubnet=%dynamic[udp/4789]
898 right=%any
899 type=transport
900 authby=psk
901 auto=route
902
903 conn input
904 leftsubnet=%dynamic[udp/4789]
905 type=transport
906 authby=psk
907 auto=route
908 ----
909
910 Generate a pre-shared key with:
911
912 ----
913 openssl rand -base64 128
914 ----
915
916 and add the key to `/etc/ipsec.secrets`, so that the file contents looks like:
917
918 ----
919 : PSK <generatedbase64key>
920 ----
921
922 Copy the PSK and the configuration to all nodes participating in the VXLAN network.