]>
Commit | Line | Data |
---|---|---|
1556b768 | 1 | [[chapter_pvesdn]] |
3c126a21 | 2 | Software-Defined Network |
1556b768 AD |
3 | ======================== |
4 | ifndef::manvolnum[] | |
5 | :pve-toplevel: | |
6 | endif::manvolnum[] | |
7 | ||
fc4bf68a TL |
8 | The **S**oftware-**D**efined **N**etwork (SDN) feature in {pve} enables the |
9 | creation of virtual zones and networks (VNets). This functionality simplifies | |
10 | advanced networking configurations and multitenancy setup." | |
1556b768 | 11 | |
fc4bf68a TL |
12 | [[pvesdn_overview]] |
13 | Introduction | |
14 | ------------ | |
15 | ||
16 | The {pve} SDN allows for separation and fine-grained control of virtual guest | |
17 | networks, using flexible, software-controlled configurations. | |
18 | ||
19 | Separation is managed through *zones*, virtual networks (*VNets*), and | |
20 | *subnets*. A zone is its own virtually separated network area. A VNet is a | |
21 | virtual network that belongs to a zone. A subnet is an IP range inside a VNet. | |
22 | ||
23 | Depending on the type of the zone, the network behaves differently and offers | |
24 | specific features, advantages, and limitations. | |
25 | ||
26 | Use cases for SDN range from an isolated private network on each individual node | |
27 | to complex overlay networks across multiple PVE clusters on different locations. | |
28 | ||
29 | After configuring an VNet in the cluster-wide datacenter SDN administration | |
30 | interface, it is available as a common Linux bridge, locally on each node, to be | |
31 | assigned to VMs and Containers. | |
32 | ||
33 | ||
34 | [[pvesdn_support_status]] | |
35 | Support Status | |
36 | -------------- | |
37 | ||
38 | History | |
39 | ~~~~~~~ | |
40 | ||
41 | The {pve} SDN stack has been available as an experimental feature since 2019 and | |
42 | has been continuously improved and tested by many developers and users. | |
43 | With its integration into the web interface in {pve} 6.2, a significant | |
44 | milestone towards broader integration was achieved. | |
45 | During the {pve} 7 release cycle, numerous improvements and features were added. | |
46 | Based on user feedback, it became apparent that the fundamental design choices | |
47 | and their implementation were quite sound and stable. Consequently, labeling it | |
48 | as `experimental' did not do justice to the state of the SDN stack. | |
49 | For {pve} 8, a decision was made to lay the groundwork for full integration of | |
50 | the SDN feature by elevating the management of networks and interfaces to a core | |
51 | component in the {pve} access control stack. | |
52 | In {pve} 8.1, two major milestones were achieved: firstly, DHCP integration was | |
53 | added to the IP address management (IPAM) feature, and secondly, the SDN | |
54 | integration is now installed by default. | |
55 | ||
56 | Current Status | |
57 | ~~~~~~~~~~~~~~ | |
ee6e18c4 | 58 | |
fc4bf68a TL |
59 | The current support status for the various layers of our SDN installation is as |
60 | follows: | |
61 | ||
62 | - Core SDN, which includes VNet management and its integration with the {pve} | |
63 | stack, is fully supported. | |
64 | - IPAM, including DHCP management for virtual guests, is in tech preview. | |
65 | - Complex routing via FRRouting and controller integration are in tech preview. | |
ee6e18c4 | 66 | |
4e652aba | 67 | [[pvesdn_installation]] |
ee6e18c4 TL |
68 | Installation |
69 | ------------ | |
70 | ||
fc4bf68a TL |
71 | SDN Core |
72 | ~~~~~~~~ | |
73 | ||
74 | Since {pve} 8.1 the core Software-Defined Network (SDN) packages are installed | |
75 | by default. | |
76 | ||
77 | If you upgrade from an older version, you need to install the | |
78 | `libpve-network-perl` package on every node: | |
1556b768 AD |
79 | |
80 | ---- | |
684db7e3 | 81 | apt update |
bcb72b25 | 82 | apt install libpve-network-perl |
1556b768 AD |
83 | ---- |
84 | ||
fc4bf68a | 85 | NOTE: {pve} version 7.0 and above have the `ifupdown2` package installed by |
bcb72b25 SL |
86 | default. If you originally installed your system with an older version, you need |
87 | to explicitly install the `ifupdown2` package. | |
fc4bf68a | 88 | + |
bcb72b25 | 89 | After installation, you need to add the following line to the end of the |
5899fa0e DW |
90 | `/etc/network/interfaces` configuration file, so that the SDN configuration gets |
91 | included and activated. | |
fc4bf68a | 92 | + |
a6af82c7 AD |
93 | ---- |
94 | source /etc/network/interfaces.d/* | |
95 | ---- | |
a6af82c7 | 96 | |
fc4bf68a TL |
97 | DHCP IPAM |
98 | ~~~~~~~~~ | |
a6af82c7 | 99 | |
fc4bf68a TL |
100 | The DHCP integration into the IP Address Management stack currently uses |
101 | `dnsmasq` for giving out DHCP leases. This is currently opt-in. | |
ee6e18c4 | 102 | |
fc4bf68a | 103 | To use that feature you need to install the `dnsmasq` package on every node: |
ee6e18c4 | 104 | |
fc4bf68a TL |
105 | ---- |
106 | apt update | |
107 | apt install dnsmasq | |
108 | ---- | |
bcb72b25 | 109 | |
fc4bf68a TL |
110 | FRRouting |
111 | ~~~~~~~~~ | |
bcb72b25 | 112 | |
fc4bf68a TL |
113 | The {pve} SDN stack uses the https://frrouting.org/[FRRouting] project for |
114 | advanced setups. This is currently opt-in. | |
ee6e18c4 | 115 | |
fc4bf68a TL |
116 | To use the SDN routing integration you need to install the `frr-pythontools` |
117 | package on all nodes: | |
ee6e18c4 | 118 | |
fc4bf68a TL |
119 | ---- |
120 | apt update | |
121 | apt install frr-pythontools | |
122 | ---- | |
bcb72b25 SL |
123 | |
124 | [[pvesdn_main_configuration]] | |
fc4bf68a TL |
125 | Configuration Overview |
126 | ---------------------- | |
1556b768 | 127 | |
460fa620 TL |
128 | Configuration is done at the web UI at datacenter level, separated into the |
129 | following sections: | |
1556b768 | 130 | |
bcb72b25 SL |
131 | * SDN:: Here you get an overview of the current active SDN state, and you can |
132 | apply all pending changes to the whole cluster. | |
1556b768 | 133 | |
bcb72b25 SL |
134 | * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated |
135 | network zones | |
a6af82c7 | 136 | |
7823f52e | 137 | * xref:pvesdn_config_vnet[VNets] VNets: Create virtual network bridges and |
bcb72b25 | 138 | manage subnets |
1556b768 | 139 | |
bcb72b25 SL |
140 | The Options category allows adding and managing additional services to be used |
141 | in your SDN setup. | |
1556b768 | 142 | |
bcb72b25 SL |
143 | * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing |
144 | in complex setups | |
a6af82c7 | 145 | |
a7ec9f05 SH |
146 | * DHCP: Define a DHCP server for a zone that automatically allocates IPs for |
147 | guests in the IPAM and leases them to the guests via DHCP. | |
148 | ||
bcb72b25 SL |
149 | * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for |
150 | guests | |
a6af82c7 | 151 | |
bcb72b25 | 152 | * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering |
fc4bf68a | 153 | virtual guests' hostname and IP addresses |
1556b768 | 154 | |
460fa620 TL |
155 | [[pvesdn_tech_and_config_overview]] |
156 | Technology & Configuration | |
fc4bf68a | 157 | -------------------------- |
460fa620 TL |
158 | |
159 | The {pve} Software-Defined Network implementation uses standard Linux networking | |
160 | as much as possible. The reason for this is that modern Linux networking | |
161 | provides almost all needs for a feature full SDN implementation and avoids adding | |
162 | external dependencies and reduces the overall amount of components that can | |
163 | break. | |
164 | ||
165 | The {pve} SDN configurations are located in `/etc/pve/sdn`, which is shared with | |
166 | all other cluster nodes through the {pve} xref:chapter_pmxcfs[configuration file system]. | |
167 | Those configurations get translated to the respective configuration formats of | |
168 | the tools that manage the underlying network stack (for example `ifupdown2` or | |
169 | `frr`). | |
170 | ||
171 | New changes are not immediately applied but recorded as pending first. You can | |
172 | then apply a set of different changes all at once in the main 'SDN' overview | |
173 | panel on the web interface. This system allows to roll-out various changes as | |
174 | single atomic one. | |
1556b768 | 175 | |
460fa620 TL |
176 | The SDN tracks the rolled-out state through the '.running-config' and '.version' |
177 | files located in '/etc/pve/sdn'. | |
1556b768 | 178 | |
460fa620 | 179 | // TODO: extend implementation and technology details. |
1556b768 | 180 | |
4e652aba | 181 | [[pvesdn_config_zone]] |
1556b768 | 182 | Zones |
a6af82c7 | 183 | ----- |
1556b768 | 184 | |
bcb72b25 | 185 | A zone defines a virtually separated network. Zones are restricted to |
5899fa0e DW |
186 | specific nodes and assigned permissions, in order to restrict users to a certain |
187 | zone and its contained VNets. | |
1556b768 | 188 | |
5899fa0e | 189 | Different technologies can be used for separation: |
1556b768 | 190 | |
bcb72b25 SL |
191 | * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT) |
192 | ||
5899fa0e | 193 | * VLAN: Virtual LANs are the classic method of subdividing a LAN |
1556b768 | 194 | |
5899fa0e | 195 | * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`) |
1556b768 | 196 | |
bcb72b25 | 197 | * VXLAN: Layer 2 VXLAN network via a UDP tunnel |
a6af82c7 | 198 | |
bcb72b25 | 199 | * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing |
1556b768 | 200 | |
1556b768 | 201 | |
bcb72b25 SL |
202 | [[pvesdn_config_common_options]] |
203 | Common Options | |
a6af82c7 | 204 | ~~~~~~~~~~~~~~ |
1556b768 | 205 | |
5899fa0e | 206 | The following options are available for all zone types: |
3093e62d | 207 | |
bcb72b25 | 208 | Nodes:: The nodes which the zone and associated VNets should be deployed on. |
9e773815 | 209 | |
bcb72b25 SL |
210 | IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the |
211 | zone. Optional, defaults to `pve`. | |
1556b768 | 212 | |
bcb72b25 | 213 | DNS:: DNS API server. Optional. |
1556b768 | 214 | |
bcb72b25 | 215 | ReverseDNS:: Reverse DNS API server. Optional. |
1556b768 | 216 | |
bcb72b25 SL |
217 | DNSZone:: DNS domain name. Used to register hostnames, such as |
218 | `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional. | |
1556b768 AD |
219 | |
220 | ||
a6af82c7 AD |
221 | [[pvesdn_zone_plugin_simple]] |
222 | Simple Zones | |
223 | ~~~~~~~~~~~~ | |
1556b768 | 224 | |
bcb72b25 SL |
225 | This is the simplest plugin. It will create an isolated VNet bridge. This |
226 | bridge is not linked to a physical interface, and VM traffic is only local on | |
227 | each the node. | |
228 | It can be used in NAT or routed setups. | |
229 | ||
1556b768 | 230 | |
4e652aba | 231 | [[pvesdn_zone_plugin_vlan]] |
ee6e18c4 TL |
232 | VLAN Zones |
233 | ~~~~~~~~~~ | |
1556b768 | 234 | |
bcb72b25 SL |
235 | The VLAN plugin uses an existing local Linux or OVS bridge to connect to the |
236 | node's physical interface. It uses VLAN tagging defined in the VNet to isolate | |
237 | the network segments. This allows connectivity of VMs between different nodes. | |
238 | ||
239 | VLAN zone configuration options: | |
1556b768 | 240 | |
bcb72b25 SL |
241 | Bridge:: The local bridge or OVS switch, already configured on *each* node that |
242 | allows node-to-node connection. | |
1556b768 | 243 | |
1556b768 | 244 | |
4e652aba | 245 | [[pvesdn_zone_plugin_qinq]] |
ee6e18c4 TL |
246 | QinQ Zones |
247 | ~~~~~~~~~~ | |
1556b768 | 248 | |
bcb72b25 SL |
249 | QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for |
250 | isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN') | |
251 | whereas the inner VLAN tag is defined by the VNet. | |
1556b768 | 252 | |
5899fa0e | 253 | NOTE: Your physical network switches must support stacked VLANs for this |
bcb72b25 | 254 | configuration. |
1556b768 | 255 | |
bcb72b25 | 256 | QinQ zone configuration options: |
1556b768 | 257 | |
bcb72b25 | 258 | Bridge:: A local, VLAN-aware bridge that is already configured on each local |
5899fa0e | 259 | node |
4e652aba | 260 | |
bcb72b25 | 261 | Service VLAN:: The main VLAN tag of this zone |
4e652aba | 262 | |
bcb72b25 | 263 | Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or |
5899fa0e | 264 | 802.1ad service VLAN type. |
9a4f3c95 | 265 | |
bcb72b25 | 266 | MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs. |
5899fa0e DW |
267 | For example, you must reduce the MTU to `1496` if you physical interface MTU is |
268 | `1500`. | |
1556b768 | 269 | |
bcb72b25 | 270 | |
4e652aba | 271 | [[pvesdn_zone_plugin_vxlan]] |
ee6e18c4 TL |
272 | VXLAN Zones |
273 | ~~~~~~~~~~~ | |
1556b768 | 274 | |
bcb72b25 SL |
275 | The VXLAN plugin establishes a tunnel (overlay) on top of an existing network |
276 | (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP | |
277 | datagrams using the default destination port `4789`. | |
278 | ||
279 | You have to configure the underlay network yourself to enable UDP connectivity | |
280 | between all peers. | |
1556b768 | 281 | |
bcb72b25 SL |
282 | You can, for example, create a VXLAN overlay network on top of public internet, |
283 | appearing to the VMs as if they share the same local Layer 2 network. | |
5899fa0e | 284 | |
bcb72b25 SL |
285 | WARNING: VXLAN on its own does does not provide any encryption. When joining |
286 | multiple sites via VXLAN, make sure to establish a secure connection between | |
287 | the site, for example by using a site-to-site VPN. | |
1556b768 | 288 | |
bcb72b25 | 289 | VXLAN zone configuration options: |
1556b768 | 290 | |
bcb72b25 SL |
291 | Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This |
292 | can be external nodes reachable at this IP address. | |
293 | All nodes in the cluster need to be mentioned here. | |
4e652aba | 294 | |
bcb72b25 | 295 | MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes |
5899fa0e | 296 | lower than the outgoing physical interface. |
1556b768 | 297 | |
bcb72b25 | 298 | |
4e652aba | 299 | [[pvesdn_zone_plugin_evpn]] |
ee6e18c4 TL |
300 | EVPN Zones |
301 | ~~~~~~~~~~ | |
1556b768 | 302 | |
bcb72b25 SL |
303 | The EVPN zone creates a routable Layer 3 network, capable of spanning across |
304 | multiple clusters. This is achieved by establishing a VPN and utilizing BGP as | |
305 | the routing protocol. | |
1556b768 | 306 | |
bcb72b25 SL |
307 | The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge |
308 | IP is the same on each node, meaning a virtual guest can use this address as | |
309 | gateway. | |
1556b768 | 310 | |
ee6e18c4 TL |
311 | Routing can work across VNets from different zones through a VRF (Virtual |
312 | Routing and Forwarding) interface. | |
1556b768 | 313 | |
bcb72b25 | 314 | EVPN zone configuration options: |
1556b768 | 315 | |
bcb72b25 | 316 | VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets. |
5899fa0e | 317 | It must be different than the VXLAN-ID of the VNets. |
1556b768 | 318 | |
bcb72b25 | 319 | Controller:: The EVPN-controller to use for this zone. (See controller plugins |
5899fa0e | 320 | section). |
1556b768 | 321 | |
bcb72b25 SL |
322 | VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this |
323 | zone. Will be auto-generated if not defined. | |
a6af82c7 | 324 | |
bcb72b25 SL |
325 | Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN |
326 | network, through the real network. The configured nodes will announce a | |
327 | default route in the EVPN network. Optional. | |
a6af82c7 | 328 | |
bcb72b25 SL |
329 | Primary Exit Node:: If you use multiple exit nodes, force traffic through this |
330 | primary exit node, instead of load-balancing on all nodes. Optional but | |
331 | necessary if you want to use SNAT or if your upstream router doesn't support | |
5899fa0e | 332 | ECMP. |
f4e692a3 | 333 | |
bcb72b25 SL |
334 | Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT |
335 | service from an exit node. (By default, the exit nodes only allow forwarding | |
336 | traffic between real network and EVPN network). Optional. | |
f4e692a3 | 337 | |
bcb72b25 SL |
338 | Advertise Subnets:: Announce the full subnet in the EVPN network. |
339 | If you have silent VMs/CTs (for example, if you have multiple IPs and the | |
340 | anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be | |
341 | able to be reached inside the EVPN network). Optional. | |
4eb24270 | 342 | |
bcb72b25 SL |
343 | Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery) |
344 | packets. This is required if you use floating IPs in your VMs (IP and MAC | |
345 | addresses are being moved between systems). Optional. | |
f4e692a3 | 346 | |
bcb72b25 SL |
347 | Route-target Import:: Allows you to import a list of external EVPN route |
348 | targets. Used for cross-DC or different EVPN network interconnects. Optional. | |
4eb24270 | 349 | |
5899fa0e | 350 | MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes |
bcb72b25 SL |
351 | less than the maximal MTU of the outgoing physical interface. Optional, |
352 | defaults to 1450. | |
1556b768 AD |
353 | |
354 | ||
7823f52e | 355 | [[pvesdn_config_vnet]] |
a6af82c7 AD |
356 | VNets |
357 | ----- | |
358 | ||
bcb72b25 SL |
359 | After creating a virtual network (VNet) through the SDN GUI, a local network |
360 | interface with the same name is available on each node. To connect a guest to the | |
361 | VNet, assign the interface to the guest and set the IP address accordingly. | |
a6af82c7 | 362 | |
bcb72b25 SL |
363 | Depending on the zone, these options have different meanings and are explained |
364 | in the respective zone section in this document. | |
a6af82c7 | 365 | |
bcb72b25 SL |
366 | WARNING: In the current state, some options may have no effect or won't work in |
367 | certain zones. | |
a6af82c7 | 368 | |
bcb72b25 SL |
369 | VNet configuration options: |
370 | ||
371 | ID:: An up to 8 character ID to identify a VNet | |
372 | ||
373 | Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional | |
a6af82c7 AD |
374 | |
375 | Zone:: The associated zone for this VNet | |
376 | ||
5899fa0e | 377 | Tag:: The unique VLAN or VXLAN ID |
a6af82c7 | 378 | |
bcb72b25 SL |
379 | VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration |
380 | in the quest. | |
381 | ||
a6af82c7 AD |
382 | |
383 | [[pvesdn_config_subnet]] | |
5899fa0e | 384 | Subnets |
bcb72b25 | 385 | ------- |
3093e62d | 386 | |
bcb72b25 SL |
387 | A subnet define a specific IP range, described by the CIDR network address. |
388 | Each VNet, can have one or more subnets. | |
1556b768 | 389 | |
3093e62d | 390 | A subnet can be used to: |
a6af82c7 | 391 | |
5899fa0e DW |
392 | * Restrict the IP addresses you can define on a specific VNet |
393 | * Assign routes/gateways on a VNet in layer 3 zones | |
394 | * Enable SNAT on a VNet in layer 3 zones | |
395 | * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins | |
3093e62d | 396 | * DNS registration through DNS plugins |
a6af82c7 | 397 | |
5899fa0e | 398 | If an IPAM server is associated with the subnet zone, the subnet prefix will be |
3093e62d | 399 | automatically registered in the IPAM. |
a6af82c7 | 400 | |
bcb72b25 | 401 | Subnet configuration options: |
a6af82c7 | 402 | |
5899fa0e | 403 | ID:: A CIDR network address, for example 10.0.0.0/8 |
a6af82c7 | 404 | |
5899fa0e DW |
405 | Gateway:: The IP address of the network's default gateway. On layer 3 zones |
406 | (Simple/EVPN plugins), it will be deployed on the VNet. | |
3093e62d | 407 | |
bcb72b25 SL |
408 | SNAT:: Enable Source NAT which allows VMs from inside a |
409 | VNet to connect to the outside network by forwarding the packets to the nodes | |
410 | outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes. | |
411 | Optional. | |
412 | ||
413 | DNS Zone Prefix:: Add a prefix to the domain registration, like | |
414 | <hostname>.prefix.<domain> Optional. | |
a6af82c7 AD |
415 | |
416 | ||
a6af82c7 AD |
417 | [[pvesdn_config_controllers]] |
418 | Controllers | |
419 | ----------- | |
420 | ||
bcb72b25 SL |
421 | Some zones implement a separated control and data plane that require an external |
422 | external controller to manage the VNet's control plane. | |
423 | ||
424 | Currently, only the `EVPN` zone requires an external controller. | |
425 | ||
8ac25ffe | 426 | |
4e652aba | 427 | [[pvesdn_controller_plugin_evpn]] |
ee6e18c4 TL |
428 | EVPN Controller |
429 | ~~~~~~~~~~~~~~~ | |
1556b768 | 430 | |
bcb72b25 SL |
431 | The `EVPN`, zone requires an external controller to manage the control plane. |
432 | The EVPN controller plugin configures the Free Range Routing (frr) router. | |
433 | ||
434 | To enable the EVPN controller, you need to install frr on every node that shall | |
435 | participate in the EVPN zone. | |
1556b768 AD |
436 | |
437 | ---- | |
a6af82c7 | 438 | apt install frr frr-pythontools |
1556b768 AD |
439 | ---- |
440 | ||
bcb72b25 | 441 | EVPN controller configuration options: |
1556b768 | 442 | |
bcb72b25 | 443 | ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN |
5899fa0e DW |
444 | number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up |
445 | breaking global routing by mistake. | |
1556b768 | 446 | |
bcb72b25 SL |
447 | Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be |
448 | external nodes or route reflector servers) | |
1556b768 | 449 | |
1556b768 | 450 | |
a6af82c7 AD |
451 | [[pvesdn_controller_plugin_BGP]] |
452 | BGP Controller | |
bcb72b25 | 453 | ~~~~~~~~~~~~~~ |
1556b768 | 454 | |
5899fa0e DW |
455 | The BGP controller is not used directly by a zone. |
456 | You can use it to configure FRR to manage BGP peers. | |
1556b768 | 457 | |
5899fa0e | 458 | For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP. |
065b2147 | 459 | It can also be used to export EVPN routes to an external BGP peer. |
4cf52a0c | 460 | |
065b2147 FG |
461 | NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP |
462 | controller. | |
1556b768 | 463 | |
bcb72b25 | 464 | BGP controller configuration options: |
1556b768 | 465 | |
bcb72b25 | 466 | Node:: The node of this BGP controller |
9a4f3c95 | 467 | |
bcb72b25 | 468 | ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN |
5899fa0e DW |
469 | number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise |
470 | you could break global routing by mistake. | |
a6af82c7 | 471 | |
bcb72b25 | 472 | Peer:: A list of peer IP addresses you want to communicate with using the |
5899fa0e | 473 | underlying BGP network. |
a6af82c7 | 474 | |
bcb72b25 | 475 | EBGP:: If your peer's remote-AS is different, this enables EBGP. |
a6af82c7 | 476 | |
bcb72b25 | 477 | Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network |
5899fa0e | 478 | (for multipath). |
a6af82c7 | 479 | |
5899fa0e DW |
480 | ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are |
481 | not directly connected or they use loopback. | |
a6af82c7 | 482 | |
5899fa0e | 483 | bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN. |
f4e692a3 | 484 | |
e5900013 AD |
485 | |
486 | [[pvesdn_controller_plugin_ISIS]] | |
487 | ISIS Controller | |
488 | ~~~~~~~~~~~~~~~ | |
489 | ||
490 | The ISIS controller is not used directly by a zone. | |
bcb72b25 | 491 | You can use it to configure FRR to export EVPN routes to an ISIS domain. |
e5900013 | 492 | |
bcb72b25 | 493 | ISIS controller configuration options: |
e5900013 | 494 | |
bcb72b25 | 495 | Node:: The node of this ISIS controller. |
e5900013 | 496 | |
bcb72b25 | 497 | Domain:: A unique ISIS domain. |
e5900013 | 498 | |
bcb72b25 | 499 | Network Entity Title:: A Unique ISIS network address that identifies this node. |
e5900013 | 500 | |
bcb72b25 | 501 | Interfaces:: A list of physical interface(s) used by ISIS. |
e5900013 | 502 | |
bcb72b25 | 503 | Loopback:: Use a loopback or dummy interface as the source of the EVPN network |
e5900013 AD |
504 | (for multipath). |
505 | ||
bcb72b25 | 506 | |
a6af82c7 | 507 | [[pvesdn_config_ipam]] |
bcb72b25 SL |
508 | IPAM |
509 | ---- | |
510 | ||
511 | IP Address Management (IPAM) tools manage the IP addresses of clients on the | |
512 | network. SDN in {pve} uses IPAM for example to find free IP addresses for new | |
513 | guests. | |
a6af82c7 | 514 | |
bcb72b25 | 515 | A single IPAM instance can be associated with one or more zones. |
a6af82c7 AD |
516 | |
517 | ||
518 | [[pvesdn_ipam_plugin_pveipam]] | |
bcb72b25 SL |
519 | PVE IPAM Plugin |
520 | ~~~~~~~~~~~~~~~ | |
a6af82c7 | 521 | |
bcb72b25 | 522 | The default built-in IPAM for your {pve} cluster. |
a6af82c7 | 523 | |
a7ec9f05 SH |
524 | You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in |
525 | the SDN section of the datacenter configuration. This UI can be used to create, | |
526 | update and delete IP mappings. This is particularly convenient in conjunction | |
527 | with the xref:pvesdn_config_dhcp[DHCP feature]. | |
528 | ||
529 | If you are using DHCP, you can use the IPAM panel to create or edit leases for | |
530 | specific VMs, which enables you to change the IPs allocated via DHCP. When | |
531 | editing an IP of a VM that is using DHCP you must make sure to force the guest | |
532 | to acquire a new DHCP leases. This can usually be done by reloading the network | |
533 | stack of the guest or rebooting it. | |
a6af82c7 | 534 | |
bcb72b25 SL |
535 | [[pvesdn_ipam_plugin_netbox]] |
536 | NetBox IPAM Plugin | |
537 | ~~~~~~~~~~~~~ | |
a6af82c7 | 538 | |
bcb72b25 SL |
539 | link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP |
540 | Address Management (IPAM) and datacenter infrastructure management (DCIM) tool. | |
a6af82c7 | 541 | |
bcb72b25 SL |
542 | To integrate NetBox with {pve} SDN, create an API token in NetBox as described |
543 | here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens | |
5899fa0e | 544 | |
bcb72b25 | 545 | The NetBox configuration properties are: |
5899fa0e | 546 | |
bcb72b25 | 547 | URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api` |
a6af82c7 | 548 | |
bcb72b25 | 549 | Token:: An API access token |
3093e62d | 550 | |
a6af82c7 | 551 | |
bcb72b25 SL |
552 | [[pvesdn_ipam_plugin_phpipam]] |
553 | phpIPAM Plugin | |
554 | ~~~~~~~~~~~~~~ | |
a6af82c7 | 555 | |
bcb72b25 SL |
556 | In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add |
557 | an API token with admin privileges to the application. | |
558 | ||
559 | The phpIPAM configuration properties are: | |
a6af82c7 | 560 | |
bcb72b25 SL |
561 | URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/` |
562 | ||
563 | Token:: An API access token | |
564 | ||
565 | Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default | |
566 | installations use `sectionid=1` for customers. | |
5899fa0e | 567 | |
a6af82c7 AD |
568 | |
569 | [[pvesdn_config_dns]] | |
3093e62d | 570 | DNS |
a6af82c7 | 571 | --- |
3093e62d TL |
572 | |
573 | The DNS plugin in {pve} SDN is used to define a DNS API server for registration | |
5899fa0e DW |
574 | of your hostname and IP address. A DNS configuration is associated with one or |
575 | more zones, to provide DNS registration for all the subnet IPs configured for | |
3093e62d | 576 | a zone. |
a6af82c7 AD |
577 | |
578 | [[pvesdn_dns_plugin_powerdns]] | |
5899fa0e | 579 | PowerDNS Plugin |
a6af82c7 AD |
580 | ~~~~~~~~~~~~~~~ |
581 | https://doc.powerdns.com/authoritative/http-api/index.html | |
582 | ||
5899fa0e | 583 | You need to enable the web server and the API in your PowerDNS config: |
1556b768 | 584 | |
40f4dfc2 | 585 | ---- |
a6af82c7 AD |
586 | api=yes |
587 | api-key=arandomgeneratedstring | |
588 | webserver=yes | |
589 | webserver-port=8081 | |
40f4dfc2 | 590 | ---- |
40f4dfc2 | 591 | |
5899fa0e | 592 | The PowerDNS configuration options are: |
a6af82c7 | 593 | |
3093e62d | 594 | url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost |
5899fa0e | 595 | |
3093e62d | 596 | key:: An API access key |
5899fa0e | 597 | |
3093e62d | 598 | ttl:: The default TTL for records |
1556b768 | 599 | |
1556b768 | 600 | |
a7ec9f05 SH |
601 | [[pvesdn_config_dhcp]] |
602 | DHCP | |
603 | ------ | |
604 | ||
605 | The DHCP plugin in {pve} SDN can be used to automatically deploy a DHCP server | |
606 | for a Zone. It provides DHCP for all Subnets in a Zone that have a DHCP range | |
607 | configured. Currently the only available backend plugin for DHCP is the dnsmasq | |
608 | plugin. | |
609 | ||
610 | The DHCP plugin works by allocating an IP in the IPAM plugin configured in the | |
611 | Zone when adding a new network interface to a VM/CT. You can find more | |
612 | information on how to configure an IPAM in the | |
613 | xref:pvesdn_config_ipam[respective section of our documentation]. | |
614 | ||
615 | When the VM starts, a mapping for the MAC address and IP gets created in the DHCP | |
616 | plugin of the zone. When the network interfaces is removed or the VM/CT are | |
617 | destroyed, then the entry in the IPAM and the DHCP server are deleted as well. | |
618 | ||
619 | NOTE: Some features (adding/editing/removing IP mappings) are currently only | |
620 | available when using the xref:pvesdn_ipam_plugin_pveipam[PVE IPAM plugin]. | |
621 | ||
622 | ||
623 | Configuration | |
624 | ~~~~~~~~~~~~~ | |
625 | ||
626 | You can enable automatic DHCP for a zone in the Web UI via the Zones panel and | |
627 | enabling DHCP in the advanced options of a zone. | |
628 | ||
629 | NOTE: Currently only Simple Zones have support for automatic DHCP | |
630 | ||
631 | After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be | |
632 | configured for the subnets in a Zone. In order to that, go to the Vnets panel and | |
633 | select the Subnet for which you want to configure DHCP ranges. In the edit | |
634 | dialogue you can configure DHCP ranges in the respective Tab. Alternatively you | |
635 | can set DHCP ranges for a Subnet via the following CLI command: | |
636 | ||
637 | ---- | |
638 | pvesh set /cluster/sdn/vnets/<vnet>/subnets/<subnet> | |
639 | -dhcp-range start-address=10.0.1.100,end-address=10.0.1.200 | |
640 | -dhcp-range start-address=10.0.2.100,end-address=10.0.2.200 | |
641 | ---- | |
642 | ||
643 | You also need to have a gateway configured for the subnet - otherwise | |
644 | automatic DHCP will not work. | |
645 | ||
646 | The DHCP plugin will then allocate IPs in the IPAM only in the configured | |
647 | ranges. | |
648 | ||
649 | Do not forget to follow the installation steps for the | |
650 | xref:pvesdn_dhcp_dnsmasq_installation[dnsmasq DHCP plugin] as well. | |
651 | ||
652 | Plugins | |
653 | ~~~~~~~ | |
654 | ||
655 | Dnsmasq Plugin | |
656 | ^^^^^^^^^^^^^^ | |
657 | Currently this is the only DHCP plugin and therefore the plugin that gets used | |
658 | when you enable DHCP for a zone. | |
659 | ||
660 | [[pvesdn_dhcp_dnsmasq_installation]] | |
661 | .Installation | |
662 | In order to be able to use the Dnsmasq plugin you need to install | |
663 | the dnsmasq package and disable the default DNS server that gets automatically | |
664 | started: | |
665 | ||
666 | ---- | |
667 | apt install dnsmasq | |
668 | systemctl disable --now dnsmasq | |
669 | ---- | |
670 | ||
671 | .Configuration | |
672 | The plugin will create a new systemd service for each zone that dnsmasq gets | |
673 | deployed to. The name for the service is `dnsmasq@<zone>`. The lifecycle of this | |
674 | service is managed by the DHCP plugin. | |
675 | ||
676 | The plugin automatically generates the following configuration files in the | |
677 | folder `/etc/dnsmasq.d/<zone>`: | |
678 | ||
679 | `00-default.conf`:: | |
680 | This contains the default global configuration for a dnsmasq instance. | |
681 | ||
682 | `10-<zone>-<subnet_cidr>.conf`:: | |
683 | This file configures specific options for a subnet, such as the DNS server that | |
684 | should get configured via DHCP. | |
685 | ||
686 | `10-<zone>-<subnet_cidr>.ranges.conf`:: | |
687 | This file configures the DHCP ranges for the dnsmasq instance. | |
688 | ||
689 | `ethers`:: | |
690 | This file contains the MAC-address and IP mappings from the IPAM plugin. In | |
691 | order to override those mappings, please use the respective IPAM plugin rather | |
692 | than editing this file, as it will get overwritten by the dnsmasq plugin. | |
693 | ||
694 | You must not edit any of the above files, since they are managed by the DHCP | |
695 | plugin. In order to customize the dnsmasq configuration you can create | |
696 | additional files (e.g. `90-custom.conf`) in the configuration folder - they will | |
697 | not get changed by the dnsmasq DHCP plugin. | |
698 | ||
699 | Configuration files are read in order, so you can control the order of the | |
700 | configuration directives by naming your custom configuration files appropriately. | |
701 | ||
702 | DHCP leases are stored in the file `/var/lib/misc/dnsmasq.<zone>.leases`. | |
703 | ||
704 | When using the PVE IPAM plugin, you can update, create and delete DHCP leases. | |
705 | For more information please consult the documentation of | |
706 | xref:pvesdn_ipam_plugin_pveipam[the PVE IPAM plugin]. Changing DHCP leases is | |
707 | currently not supported for the other IPAM plugins. | |
708 | ||
bcb72b25 | 709 | [[pvesdn_setup_examples]] |
a6af82c7 AD |
710 | Examples |
711 | -------- | |
712 | ||
bcb72b25 SL |
713 | This section presents multiple configuration examples tailored for common SDN |
714 | use cases. It aims to offer tangible implementations, providing additional | |
715 | details to enhance comprehension of the available configuration options. | |
1556b768 | 716 | |
ee6e18c4 | 717 | |
bcb72b25 SL |
718 | [[pvesdn_setup_example_simple]] |
719 | Simple Zone Example | |
720 | ~~~~~~~~~~~~~~~~~~~ | |
1556b768 | 721 | |
bcb72b25 SL |
722 | Simple zone networks create an isolated network for quests on a single host to |
723 | connect to each other. | |
1556b768 | 724 | |
bcb72b25 SL |
725 | TIP: connection between quests are possible if all quests reside on a same host |
726 | but cannot be reached on other nodes. | |
1556b768 | 727 | |
bcb72b25 SL |
728 | * Create a simple zone named `simple`. |
729 | * Add a VNet names `vnet1`. | |
730 | * Create a Subnet with a gateway and the SNAT option enabled. | |
731 | * This creates a network bridge `vnet1` on the node. Assign this bridge to the | |
732 | quests that shall join the network and configure an IP address. | |
1556b768 | 733 | |
bcb72b25 SL |
734 | The network interface configuration in two VMs may look like this which allows |
735 | them to communicate via the 10.0.1.0/24 network. | |
1556b768 AD |
736 | |
737 | ---- | |
bcb72b25 SL |
738 | allow-hotplug ens19 |
739 | iface ens19 inet static | |
740 | address 10.0.1.14/24 | |
741 | ---- | |
1556b768 | 742 | |
bcb72b25 SL |
743 | ---- |
744 | allow-hotplug ens19 | |
745 | iface ens19 inet static | |
746 | address 10.0.1.15/24 | |
747 | ---- | |
1556b768 | 748 | |
bcb72b25 SL |
749 | |
750 | [[pvesdn_setup_example_nat]] | |
751 | Source NAT Example | |
752 | ~~~~~~~~~~~~~~~~~~ | |
753 | ||
754 | If you want to allow outgoing connections for quests in the simple network zone | |
755 | the simple zone offers a Source NAT (SNAT) option. | |
756 | ||
757 | Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a | |
758 | Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option. | |
759 | ||
760 | ---- | |
761 | Subnet: 172.16.0.0/24 | |
762 | Gateway: 172.16.0.1 | |
763 | SNAT: checked | |
1556b768 AD |
764 | ---- |
765 | ||
bcb72b25 SL |
766 | In the quests configure the static IP address inside the subnet's IP range. |
767 | ||
768 | The node itself will join this network with the Gateway IP '172.16.0.1' and | |
769 | function as the NAT gateway for quests within the subnet range. | |
770 | ||
771 | ||
772 | [[pvesdn_setup_example_vlan]] | |
773 | VLAN Setup Example | |
774 | ~~~~~~~~~~~~~~~~~~ | |
775 | ||
776 | When VMs on different nodes need to communicate through an isolated network, the | |
777 | VLAN zone allows network level isolation using VLAN tags. | |
778 | ||
779 | Create a VLAN zone named `myvlanzone`: | |
1556b768 AD |
780 | |
781 | ---- | |
bcb72b25 SL |
782 | ID: myvlanzone |
783 | Bridge: vmbr0 | |
1556b768 AD |
784 | ---- |
785 | ||
bcb72b25 SL |
786 | Create a VNet named `myvnet1` with VLAN tag 10 and the previously created |
787 | `myvlanzone`. | |
1556b768 AD |
788 | |
789 | ---- | |
bcb72b25 SL |
790 | ID: myvnet1 |
791 | Zone: myvlanzone | |
792 | Tag: 10 | |
1556b768 AD |
793 | ---- |
794 | ||
ee6e18c4 | 795 | Apply the configuration through the main SDN panel, to create VNets locally on |
5899fa0e | 796 | each node. |
1556b768 | 797 | |
bcb72b25 | 798 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`. |
1556b768 | 799 | |
ee6e18c4 | 800 | Use the following network configuration for this VM: |
1556b768 AD |
801 | |
802 | ---- | |
803 | auto eth0 | |
804 | iface eth0 inet static | |
ee6e18c4 | 805 | address 10.0.3.100/24 |
1556b768 AD |
806 | ---- |
807 | ||
bcb72b25 SL |
808 | Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet |
809 | `myvnet1` as vm1. | |
ee6e18c4 TL |
810 | |
811 | Use the following network configuration for this VM: | |
812 | ||
1556b768 AD |
813 | ---- |
814 | auto eth0 | |
815 | iface eth0 inet static | |
ee6e18c4 | 816 | address 10.0.3.101/24 |
1556b768 AD |
817 | ---- |
818 | ||
bcb72b25 | 819 | Following this, you should be able to ping between both VMs using that network. |
1556b768 AD |
820 | |
821 | ||
4e652aba TL |
822 | [[pvesdn_setup_example_qinq]] |
823 | QinQ Setup Example | |
a6af82c7 | 824 | ~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 825 | |
1556b768 | 826 | |
bcb72b25 SL |
827 | This example configures two QinQ zones and adds two VMs to each zone to |
828 | demonstrate the additional layer of VLAN tags which allows the configuration of | |
829 | more isolated VLANs. | |
1556b768 | 830 | |
bcb72b25 SL |
831 | A typical use case for this configuration is a hosting provider that provides an |
832 | isolated network to customers for VM communication but isolates the VMs from | |
833 | other customers. | |
1556b768 | 834 | |
bcb72b25 | 835 | Create a QinQ zone named `qinqzone1` with service VLAN 20 |
1556b768 AD |
836 | |
837 | ---- | |
bcb72b25 SL |
838 | ID: qinqzone1 |
839 | Bridge: vmbr0 | |
840 | Service VLAN: 20 | |
1556b768 AD |
841 | ---- |
842 | ||
bcb72b25 | 843 | Create another QinQ zone named `qinqzone2` with service VLAN 30 |
1556b768 | 844 | ---- |
bcb72b25 SL |
845 | ID: qinqzone2 |
846 | Bridge: vmbr0 | |
847 | Service VLAN: 30 | |
1556b768 AD |
848 | ---- |
849 | ||
bcb72b25 SL |
850 | Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created |
851 | `qinqzone1` zone. | |
1556b768 AD |
852 | |
853 | ---- | |
bcb72b25 SL |
854 | ID: qinqvnet1 |
855 | Zone: qinqzone1 | |
856 | Tag: 100 | |
1556b768 AD |
857 | ---- |
858 | ||
bcb72b25 | 859 | Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone. |
1556b768 AD |
860 | |
861 | ---- | |
bcb72b25 SL |
862 | ID: qinqvnet2 |
863 | Zone: qinqzone2 | |
864 | Tag: 100 | |
1556b768 AD |
865 | ---- |
866 | ||
ee6e18c4 | 867 | Apply the configuration on the main SDN web-interface panel to create VNets |
bcb72b25 | 868 | locally on each node. |
1556b768 | 869 | |
bcb72b25 SL |
870 | Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network |
871 | interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge | |
872 | `qinqvnet2`. | |
ee6e18c4 | 873 | |
bcb72b25 SL |
874 | Inside the VM, configure the IP addresses of the interfaces, for example via |
875 | `/etc/network/interfaces`: | |
ee6e18c4 | 876 | |
1556b768 AD |
877 | ---- |
878 | auto eth0 | |
879 | iface eth0 inet static | |
bcb72b25 | 880 | address 10.0.3.101/24 |
1556b768 | 881 | ---- |
bcb72b25 SL |
882 | // TODO: systemd-network example |
883 | Configure all four VMs to have IP addresses from the '10.0.3.101' to | |
884 | '10.0.3.104' range. | |
1556b768 | 885 | |
bcb72b25 | 886 | Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as |
5899fa0e | 887 | between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs |
bcb72b25 | 888 | 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN. |
1556b768 | 889 | |
1556b768 | 890 | |
4e652aba | 891 | [[pvesdn_setup_example_vxlan]] |
ee6e18c4 | 892 | VXLAN Setup Example |
a6af82c7 | 893 | ~~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 894 | |
bcb72b25 SL |
895 | The example assumes a cluster with three nodes, with the node IP addresses |
896 | 192.168.0.1, 192.168.0.2 and 192.168.0.3. | |
1556b768 | 897 | |
bcb72b25 SL |
898 | Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the |
899 | peer address list. Use the default MTU of 1450 or configure accordingly. | |
1556b768 | 900 | |
1556b768 | 901 | ---- |
bcb72b25 SL |
902 | ID: myvxlanzone |
903 | Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3 | |
1556b768 AD |
904 | ---- |
905 | ||
bcb72b25 | 906 | Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created |
ee6e18c4 | 907 | previously. |
1556b768 AD |
908 | |
909 | ---- | |
bcb72b25 SL |
910 | ID: vxvnet1 |
911 | Zone: myvxlanzone | |
912 | Tag: 100000 | |
1556b768 AD |
913 | ---- |
914 | ||
ee6e18c4 TL |
915 | Apply the configuration on the main SDN web-interface panel to create VNets |
916 | locally on each nodes. | |
1556b768 | 917 | |
bcb72b25 | 918 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`. |
1556b768 | 919 | |
5899fa0e | 920 | Use the following network configuration for this VM (note the lower MTU). |
1556b768 AD |
921 | |
922 | ---- | |
923 | auto eth0 | |
924 | iface eth0 inet static | |
bcb72b25 SL |
925 | address 10.0.3.100/24 |
926 | mtu 1450 | |
1556b768 AD |
927 | ---- |
928 | ||
bcb72b25 SL |
929 | Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet |
930 | `vxvnet1` as vm1. | |
ee6e18c4 TL |
931 | |
932 | Use the following network configuration for this VM: | |
933 | ||
1556b768 AD |
934 | ---- |
935 | auto eth0 | |
936 | iface eth0 inet static | |
bcb72b25 SL |
937 | address 10.0.3.101/24 |
938 | mtu 1450 | |
1556b768 AD |
939 | ---- |
940 | ||
ee6e18c4 | 941 | Then, you should be able to ping between between 'vm1' and 'vm2'. |
1556b768 AD |
942 | |
943 | ||
4e652aba TL |
944 | [[pvesdn_setup_example_evpn]] |
945 | EVPN Setup Example | |
a6af82c7 | 946 | ~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 947 | |
bcb72b25 SL |
948 | The example assumes a cluster with three nodes (node1, node2, node3) with IP |
949 | addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3. | |
1556b768 | 950 | |
5899fa0e DW |
951 | Create an EVPN controller, using a private ASN number and the above node |
952 | addresses as peers. | |
1556b768 AD |
953 | |
954 | ---- | |
bcb72b25 SL |
955 | ID: myevpnctl |
956 | ASN#: 65000 | |
957 | Peers: 192.168.0.1,192.168.0.2,192.168.0.3 | |
1556b768 AD |
958 | ---- |
959 | ||
bcb72b25 SL |
960 | Create an EVPN zone named `myevpnzone`, assign the previously created |
961 | EVPN-controller and define 'node1' and 'node2' as exit nodes. | |
a6af82c7 | 962 | |
1556b768 | 963 | ---- |
bcb72b25 SL |
964 | ID: myevpnzone |
965 | VRF VXLAN Tag: 10000 | |
966 | Controller: myevpnctl | |
967 | MTU: 1450 | |
968 | VNet MAC Address: 32:F4:05:FE:6C:0A | |
969 | Exit Nodes: node1,node2 | |
1556b768 AD |
970 | ---- |
971 | ||
bcb72b25 SL |
972 | Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`. |
973 | ||
1556b768 | 974 | ---- |
bcb72b25 SL |
975 | ID: myvnet1 |
976 | Zone: myevpnzone | |
977 | Tag: 11000 | |
1556b768 AD |
978 | ---- |
979 | ||
bcb72b25 | 980 | Create a subnet on `myvnet1`: |
9f819242 | 981 | |
a6af82c7 | 982 | ---- |
bcb72b25 SL |
983 | Subnet: 10.0.1.0/24 |
984 | Gateway: 10.0.1.1 | |
a6af82c7 AD |
985 | ---- |
986 | ||
bcb72b25 | 987 | Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`. |
1556b768 AD |
988 | |
989 | ---- | |
bcb72b25 SL |
990 | ID: myvnet2 |
991 | Zone: myevpnzone | |
992 | Tag: 12000 | |
1556b768 AD |
993 | ---- |
994 | ||
bcb72b25 | 995 | Create a different subnet on `myvnet2``: |
9f819242 | 996 | |
a6af82c7 | 997 | ---- |
bcb72b25 SL |
998 | Subnet: 10.0.2.0/24 |
999 | Gateway: 10.0.2.1 | |
a6af82c7 AD |
1000 | ---- |
1001 | ||
5899fa0e | 1002 | Apply the configuration from the main SDN web-interface panel to create VNets |
bcb72b25 | 1003 | locally on each node and generate the FRR configuration. |
1556b768 | 1004 | |
bcb72b25 | 1005 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`. |
1556b768 | 1006 | |
bcb72b25 | 1007 | Use the following network configuration for 'vm1': |
1556b768 AD |
1008 | |
1009 | ---- | |
1010 | auto eth0 | |
1011 | iface eth0 inet static | |
bcb72b25 SL |
1012 | address 10.0.1.100/24 |
1013 | gateway 10.0.1.1 | |
1014 | mtu 1450 | |
1556b768 AD |
1015 | ---- |
1016 | ||
bcb72b25 SL |
1017 | Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet |
1018 | `myvnet2`. | |
ee6e18c4 | 1019 | |
bcb72b25 | 1020 | Use the following network configuration for 'vm2': |
ee6e18c4 | 1021 | |
1556b768 AD |
1022 | ---- |
1023 | auto eth0 | |
1024 | iface eth0 inet static | |
bcb72b25 SL |
1025 | address 10.0.2.100/24 |
1026 | gateway 10.0.2.1 | |
1027 | mtu 1450 | |
1556b768 AD |
1028 | ---- |
1029 | ||
1030 | ||
bcb72b25 | 1031 | Now you should be able to ping vm2 from vm1, and vm1 from vm2. |
1556b768 | 1032 | |
bcb72b25 | 1033 | If you ping an external IP from 'vm2' on the non-gateway node3, the packet |
a6af82c7 | 1034 | will go to the configured 'myvnet2' gateway, then will be routed to the exit |
ee6e18c4 TL |
1035 | nodes ('node1' or 'node2') and from there it will leave those nodes over the |
1036 | default gateway configured on node1 or node2. | |
1556b768 | 1037 | |
5899fa0e DW |
1038 | NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24' |
1039 | networks to node1 and node2 on your external gateway, so that the public network | |
1040 | can reply back. | |
1556b768 | 1041 | |
ee6e18c4 TL |
1042 | If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 |
1043 | and 10.0.2.0/24 in this example), will be announced dynamically. | |
ce84ae7e AD |
1044 | |
1045 | ||
bcb72b25 | 1046 | [[pvesdn_notes]] |
ce84ae7e AD |
1047 | Notes |
1048 | ----- | |
1049 | ||
4368fce0 AD |
1050 | Multiple EVPN Exit Nodes |
1051 | ~~~~~~~~~~~~~~~~~~~~~~~~ | |
1052 | ||
5b105f3e TL |
1053 | If you have multiple gateway nodes, you should disable the `rp_filter` (Strict |
1054 | Reverse Path Filter) option, because packets can arrive at one node but go out | |
1055 | from another node. | |
4368fce0 | 1056 | |
bcb72b25 SL |
1057 | Add the following to `/etc/sysctl.conf`: |
1058 | ||
4368fce0 AD |
1059 | ----- |
1060 | net.ipv4.conf.default.rp_filter=0 | |
1061 | net.ipv4.conf.all.rp_filter=0 | |
1062 | ----- | |
1063 | ||
448c1d39 TL |
1064 | VXLAN IPSEC Encryption |
1065 | ~~~~~~~~~~~~~~~~~~~~~~ | |
5899fa0e | 1066 | |
bcb72b25 SL |
1067 | To add IPSEC encryption on top of a VXLAN, this example shows how to use |
1068 | `strongswan`. | |
1069 | ||
1070 | You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for | |
1071 | IPv6 to handle encryption. | |
ce84ae7e | 1072 | |
448c1d39 TL |
1073 | So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC) |
1074 | + 50 (VXLAN) == 1500). | |
ce84ae7e | 1075 | |
bcb72b25 SL |
1076 | Install strongswan on the host. |
1077 | ||
ce84ae7e | 1078 | ---- |
8bfa192d | 1079 | apt install strongswan |
ce84ae7e AD |
1080 | ---- |
1081 | ||
bcb72b25 | 1082 | Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from |
448c1d39 | 1083 | the VXLAN UDP port '4789'. |
ce84ae7e AD |
1084 | |
1085 | ---- | |
1086 | conn %default | |
448c1d39 | 1087 | ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW |
ce84ae7e | 1088 | esp=aes256-sha1! |
448c1d39 | 1089 | leftfirewall=yes # this is necessary when using Proxmox VE firewall rules |
ce84ae7e AD |
1090 | |
1091 | conn output | |
1092 | rightsubnet=%dynamic[udp/4789] | |
1093 | right=%any | |
1094 | type=transport | |
1095 | authby=psk | |
1096 | auto=route | |
1097 | ||
1098 | conn input | |
1099 | leftsubnet=%dynamic[udp/4789] | |
1100 | type=transport | |
1101 | authby=psk | |
1102 | auto=route | |
1103 | ---- | |
1104 | ||
bcb72b25 | 1105 | Generate a pre-shared key with: |
ce84ae7e AD |
1106 | |
1107 | ---- | |
1108 | openssl rand -base64 128 | |
1109 | ---- | |
1110 | ||
bcb72b25 | 1111 | and add the key to `/etc/ipsec.secrets`, so that the file contents looks like: |
ce84ae7e AD |
1112 | |
1113 | ---- | |
1114 | : PSK <generatedbase64key> | |
1115 | ---- | |
448c1d39 | 1116 | |
bcb72b25 | 1117 | Copy the PSK and the configuration to all nodes participating in the VXLAN network. |