]>
Commit | Line | Data |
---|---|---|
1556b768 | 1 | [[chapter_pvesdn]] |
3c126a21 | 2 | Software-Defined Network |
1556b768 AD |
3 | ======================== |
4 | ifndef::manvolnum[] | |
5 | :pve-toplevel: | |
6 | endif::manvolnum[] | |
7 | ||
fc4bf68a TL |
8 | The **S**oftware-**D**efined **N**etwork (SDN) feature in {pve} enables the |
9 | creation of virtual zones and networks (VNets). This functionality simplifies | |
2b7f3690 | 10 | advanced networking configurations and multitenancy setup. |
1556b768 | 11 | |
fc4bf68a TL |
12 | [[pvesdn_overview]] |
13 | Introduction | |
14 | ------------ | |
15 | ||
16 | The {pve} SDN allows for separation and fine-grained control of virtual guest | |
17 | networks, using flexible, software-controlled configurations. | |
18 | ||
19 | Separation is managed through *zones*, virtual networks (*VNets*), and | |
20 | *subnets*. A zone is its own virtually separated network area. A VNet is a | |
21 | virtual network that belongs to a zone. A subnet is an IP range inside a VNet. | |
22 | ||
23 | Depending on the type of the zone, the network behaves differently and offers | |
24 | specific features, advantages, and limitations. | |
25 | ||
26 | Use cases for SDN range from an isolated private network on each individual node | |
27 | to complex overlay networks across multiple PVE clusters on different locations. | |
28 | ||
29 | After configuring an VNet in the cluster-wide datacenter SDN administration | |
30 | interface, it is available as a common Linux bridge, locally on each node, to be | |
31 | assigned to VMs and Containers. | |
32 | ||
33 | ||
34 | [[pvesdn_support_status]] | |
35 | Support Status | |
36 | -------------- | |
37 | ||
38 | History | |
39 | ~~~~~~~ | |
40 | ||
41 | The {pve} SDN stack has been available as an experimental feature since 2019 and | |
42 | has been continuously improved and tested by many developers and users. | |
43 | With its integration into the web interface in {pve} 6.2, a significant | |
44 | milestone towards broader integration was achieved. | |
45 | During the {pve} 7 release cycle, numerous improvements and features were added. | |
46 | Based on user feedback, it became apparent that the fundamental design choices | |
47 | and their implementation were quite sound and stable. Consequently, labeling it | |
48 | as `experimental' did not do justice to the state of the SDN stack. | |
49 | For {pve} 8, a decision was made to lay the groundwork for full integration of | |
50 | the SDN feature by elevating the management of networks and interfaces to a core | |
51 | component in the {pve} access control stack. | |
52 | In {pve} 8.1, two major milestones were achieved: firstly, DHCP integration was | |
53 | added to the IP address management (IPAM) feature, and secondly, the SDN | |
54 | integration is now installed by default. | |
55 | ||
56 | Current Status | |
57 | ~~~~~~~~~~~~~~ | |
ee6e18c4 | 58 | |
fc4bf68a TL |
59 | The current support status for the various layers of our SDN installation is as |
60 | follows: | |
61 | ||
62 | - Core SDN, which includes VNet management and its integration with the {pve} | |
63 | stack, is fully supported. | |
64 | - IPAM, including DHCP management for virtual guests, is in tech preview. | |
65 | - Complex routing via FRRouting and controller integration are in tech preview. | |
ee6e18c4 | 66 | |
4e652aba | 67 | [[pvesdn_installation]] |
ee6e18c4 TL |
68 | Installation |
69 | ------------ | |
70 | ||
fc4bf68a TL |
71 | SDN Core |
72 | ~~~~~~~~ | |
73 | ||
74 | Since {pve} 8.1 the core Software-Defined Network (SDN) packages are installed | |
75 | by default. | |
76 | ||
77 | If you upgrade from an older version, you need to install the | |
78 | `libpve-network-perl` package on every node: | |
1556b768 AD |
79 | |
80 | ---- | |
684db7e3 | 81 | apt update |
bcb72b25 | 82 | apt install libpve-network-perl |
1556b768 AD |
83 | ---- |
84 | ||
fc4bf68a | 85 | NOTE: {pve} version 7.0 and above have the `ifupdown2` package installed by |
bcb72b25 SL |
86 | default. If you originally installed your system with an older version, you need |
87 | to explicitly install the `ifupdown2` package. | |
3d5a1655 TL |
88 | |
89 | After installation, you need to ensure that the following line is present at the | |
90 | end of the `/etc/network/interfaces` configuration file on all nodes, so that | |
91 | the SDN configuration gets included and activated. | |
92 | ||
a6af82c7 AD |
93 | ---- |
94 | source /etc/network/interfaces.d/* | |
95 | ---- | |
a6af82c7 | 96 | |
3d5a1655 | 97 | [[pvesdn_install_dhcp_ipam]] |
fc4bf68a TL |
98 | DHCP IPAM |
99 | ~~~~~~~~~ | |
a6af82c7 | 100 | |
3d5a1655 TL |
101 | The DHCP integration into the built-in 'PVE' IP Address Management stack |
102 | currently uses `dnsmasq` for giving out DHCP leases. This is currently opt-in. | |
ee6e18c4 | 103 | |
fc4bf68a | 104 | To use that feature you need to install the `dnsmasq` package on every node: |
ee6e18c4 | 105 | |
fc4bf68a TL |
106 | ---- |
107 | apt update | |
108 | apt install dnsmasq | |
3d5a1655 TL |
109 | # disable default instance |
110 | systemctl disable --now dnsmasq | |
fc4bf68a | 111 | ---- |
bcb72b25 | 112 | |
3d5a1655 | 113 | [[pvesdn_install_frrouting]] |
fc4bf68a TL |
114 | FRRouting |
115 | ~~~~~~~~~ | |
bcb72b25 | 116 | |
fc4bf68a TL |
117 | The {pve} SDN stack uses the https://frrouting.org/[FRRouting] project for |
118 | advanced setups. This is currently opt-in. | |
ee6e18c4 | 119 | |
fc4bf68a TL |
120 | To use the SDN routing integration you need to install the `frr-pythontools` |
121 | package on all nodes: | |
ee6e18c4 | 122 | |
fc4bf68a TL |
123 | ---- |
124 | apt update | |
125 | apt install frr-pythontools | |
126 | ---- | |
bcb72b25 SL |
127 | |
128 | [[pvesdn_main_configuration]] | |
fc4bf68a TL |
129 | Configuration Overview |
130 | ---------------------- | |
1556b768 | 131 | |
460fa620 TL |
132 | Configuration is done at the web UI at datacenter level, separated into the |
133 | following sections: | |
1556b768 | 134 | |
bcb72b25 SL |
135 | * SDN:: Here you get an overview of the current active SDN state, and you can |
136 | apply all pending changes to the whole cluster. | |
1556b768 | 137 | |
bcb72b25 SL |
138 | * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated |
139 | network zones | |
a6af82c7 | 140 | |
7823f52e | 141 | * xref:pvesdn_config_vnet[VNets] VNets: Create virtual network bridges and |
bcb72b25 | 142 | manage subnets |
1556b768 | 143 | |
bcb72b25 SL |
144 | The Options category allows adding and managing additional services to be used |
145 | in your SDN setup. | |
1556b768 | 146 | |
bcb72b25 SL |
147 | * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing |
148 | in complex setups | |
a6af82c7 | 149 | |
a7ec9f05 SH |
150 | * DHCP: Define a DHCP server for a zone that automatically allocates IPs for |
151 | guests in the IPAM and leases them to the guests via DHCP. | |
152 | ||
bcb72b25 SL |
153 | * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for |
154 | guests | |
a6af82c7 | 155 | |
bcb72b25 | 156 | * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering |
fc4bf68a | 157 | virtual guests' hostname and IP addresses |
1556b768 | 158 | |
460fa620 TL |
159 | [[pvesdn_tech_and_config_overview]] |
160 | Technology & Configuration | |
fc4bf68a | 161 | -------------------------- |
460fa620 TL |
162 | |
163 | The {pve} Software-Defined Network implementation uses standard Linux networking | |
164 | as much as possible. The reason for this is that modern Linux networking | |
165 | provides almost all needs for a feature full SDN implementation and avoids adding | |
166 | external dependencies and reduces the overall amount of components that can | |
167 | break. | |
168 | ||
169 | The {pve} SDN configurations are located in `/etc/pve/sdn`, which is shared with | |
170 | all other cluster nodes through the {pve} xref:chapter_pmxcfs[configuration file system]. | |
171 | Those configurations get translated to the respective configuration formats of | |
172 | the tools that manage the underlying network stack (for example `ifupdown2` or | |
173 | `frr`). | |
174 | ||
175 | New changes are not immediately applied but recorded as pending first. You can | |
176 | then apply a set of different changes all at once in the main 'SDN' overview | |
177 | panel on the web interface. This system allows to roll-out various changes as | |
178 | single atomic one. | |
1556b768 | 179 | |
460fa620 TL |
180 | The SDN tracks the rolled-out state through the '.running-config' and '.version' |
181 | files located in '/etc/pve/sdn'. | |
1556b768 | 182 | |
460fa620 | 183 | // TODO: extend implementation and technology details. |
1556b768 | 184 | |
4e652aba | 185 | [[pvesdn_config_zone]] |
1556b768 | 186 | Zones |
a6af82c7 | 187 | ----- |
1556b768 | 188 | |
bcb72b25 | 189 | A zone defines a virtually separated network. Zones are restricted to |
5899fa0e DW |
190 | specific nodes and assigned permissions, in order to restrict users to a certain |
191 | zone and its contained VNets. | |
1556b768 | 192 | |
5899fa0e | 193 | Different technologies can be used for separation: |
1556b768 | 194 | |
bcb72b25 SL |
195 | * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT) |
196 | ||
5899fa0e | 197 | * VLAN: Virtual LANs are the classic method of subdividing a LAN |
1556b768 | 198 | |
5899fa0e | 199 | * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`) |
1556b768 | 200 | |
bcb72b25 | 201 | * VXLAN: Layer 2 VXLAN network via a UDP tunnel |
a6af82c7 | 202 | |
bcb72b25 | 203 | * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing |
1556b768 | 204 | |
1556b768 | 205 | |
bcb72b25 SL |
206 | [[pvesdn_config_common_options]] |
207 | Common Options | |
a6af82c7 | 208 | ~~~~~~~~~~~~~~ |
1556b768 | 209 | |
5899fa0e | 210 | The following options are available for all zone types: |
3093e62d | 211 | |
bcb72b25 | 212 | Nodes:: The nodes which the zone and associated VNets should be deployed on. |
9e773815 | 213 | |
bcb72b25 SL |
214 | IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the |
215 | zone. Optional, defaults to `pve`. | |
1556b768 | 216 | |
bcb72b25 | 217 | DNS:: DNS API server. Optional. |
1556b768 | 218 | |
bcb72b25 | 219 | ReverseDNS:: Reverse DNS API server. Optional. |
1556b768 | 220 | |
bcb72b25 SL |
221 | DNSZone:: DNS domain name. Used to register hostnames, such as |
222 | `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional. | |
1556b768 AD |
223 | |
224 | ||
a6af82c7 AD |
225 | [[pvesdn_zone_plugin_simple]] |
226 | Simple Zones | |
227 | ~~~~~~~~~~~~ | |
1556b768 | 228 | |
bcb72b25 SL |
229 | This is the simplest plugin. It will create an isolated VNet bridge. This |
230 | bridge is not linked to a physical interface, and VM traffic is only local on | |
231 | each the node. | |
232 | It can be used in NAT or routed setups. | |
233 | ||
1556b768 | 234 | |
4e652aba | 235 | [[pvesdn_zone_plugin_vlan]] |
ee6e18c4 TL |
236 | VLAN Zones |
237 | ~~~~~~~~~~ | |
1556b768 | 238 | |
bcb72b25 SL |
239 | The VLAN plugin uses an existing local Linux or OVS bridge to connect to the |
240 | node's physical interface. It uses VLAN tagging defined in the VNet to isolate | |
241 | the network segments. This allows connectivity of VMs between different nodes. | |
242 | ||
243 | VLAN zone configuration options: | |
1556b768 | 244 | |
bcb72b25 SL |
245 | Bridge:: The local bridge or OVS switch, already configured on *each* node that |
246 | allows node-to-node connection. | |
1556b768 | 247 | |
1556b768 | 248 | |
4e652aba | 249 | [[pvesdn_zone_plugin_qinq]] |
ee6e18c4 TL |
250 | QinQ Zones |
251 | ~~~~~~~~~~ | |
1556b768 | 252 | |
bcb72b25 SL |
253 | QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for |
254 | isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN') | |
255 | whereas the inner VLAN tag is defined by the VNet. | |
1556b768 | 256 | |
5899fa0e | 257 | NOTE: Your physical network switches must support stacked VLANs for this |
bcb72b25 | 258 | configuration. |
1556b768 | 259 | |
bcb72b25 | 260 | QinQ zone configuration options: |
1556b768 | 261 | |
bcb72b25 | 262 | Bridge:: A local, VLAN-aware bridge that is already configured on each local |
5899fa0e | 263 | node |
4e652aba | 264 | |
bcb72b25 | 265 | Service VLAN:: The main VLAN tag of this zone |
4e652aba | 266 | |
bcb72b25 | 267 | Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or |
5899fa0e | 268 | 802.1ad service VLAN type. |
9a4f3c95 | 269 | |
bcb72b25 | 270 | MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs. |
5899fa0e DW |
271 | For example, you must reduce the MTU to `1496` if you physical interface MTU is |
272 | `1500`. | |
1556b768 | 273 | |
bcb72b25 | 274 | |
4e652aba | 275 | [[pvesdn_zone_plugin_vxlan]] |
ee6e18c4 TL |
276 | VXLAN Zones |
277 | ~~~~~~~~~~~ | |
1556b768 | 278 | |
bcb72b25 SL |
279 | The VXLAN plugin establishes a tunnel (overlay) on top of an existing network |
280 | (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP | |
281 | datagrams using the default destination port `4789`. | |
282 | ||
283 | You have to configure the underlay network yourself to enable UDP connectivity | |
284 | between all peers. | |
1556b768 | 285 | |
bcb72b25 SL |
286 | You can, for example, create a VXLAN overlay network on top of public internet, |
287 | appearing to the VMs as if they share the same local Layer 2 network. | |
5899fa0e | 288 | |
bcb72b25 SL |
289 | WARNING: VXLAN on its own does does not provide any encryption. When joining |
290 | multiple sites via VXLAN, make sure to establish a secure connection between | |
291 | the site, for example by using a site-to-site VPN. | |
1556b768 | 292 | |
bcb72b25 | 293 | VXLAN zone configuration options: |
1556b768 | 294 | |
bcb72b25 SL |
295 | Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This |
296 | can be external nodes reachable at this IP address. | |
297 | All nodes in the cluster need to be mentioned here. | |
4e652aba | 298 | |
bcb72b25 | 299 | MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes |
5899fa0e | 300 | lower than the outgoing physical interface. |
1556b768 | 301 | |
bcb72b25 | 302 | |
4e652aba | 303 | [[pvesdn_zone_plugin_evpn]] |
ee6e18c4 TL |
304 | EVPN Zones |
305 | ~~~~~~~~~~ | |
1556b768 | 306 | |
bcb72b25 SL |
307 | The EVPN zone creates a routable Layer 3 network, capable of spanning across |
308 | multiple clusters. This is achieved by establishing a VPN and utilizing BGP as | |
309 | the routing protocol. | |
1556b768 | 310 | |
bcb72b25 SL |
311 | The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge |
312 | IP is the same on each node, meaning a virtual guest can use this address as | |
313 | gateway. | |
1556b768 | 314 | |
ee6e18c4 TL |
315 | Routing can work across VNets from different zones through a VRF (Virtual |
316 | Routing and Forwarding) interface. | |
1556b768 | 317 | |
bcb72b25 | 318 | EVPN zone configuration options: |
1556b768 | 319 | |
bcb72b25 | 320 | VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets. |
5899fa0e | 321 | It must be different than the VXLAN-ID of the VNets. |
1556b768 | 322 | |
bcb72b25 | 323 | Controller:: The EVPN-controller to use for this zone. (See controller plugins |
5899fa0e | 324 | section). |
1556b768 | 325 | |
bcb72b25 SL |
326 | VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this |
327 | zone. Will be auto-generated if not defined. | |
a6af82c7 | 328 | |
bcb72b25 SL |
329 | Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN |
330 | network, through the real network. The configured nodes will announce a | |
331 | default route in the EVPN network. Optional. | |
a6af82c7 | 332 | |
bcb72b25 SL |
333 | Primary Exit Node:: If you use multiple exit nodes, force traffic through this |
334 | primary exit node, instead of load-balancing on all nodes. Optional but | |
335 | necessary if you want to use SNAT or if your upstream router doesn't support | |
5899fa0e | 336 | ECMP. |
f4e692a3 | 337 | |
bcb72b25 SL |
338 | Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT |
339 | service from an exit node. (By default, the exit nodes only allow forwarding | |
340 | traffic between real network and EVPN network). Optional. | |
f4e692a3 | 341 | |
bcb72b25 SL |
342 | Advertise Subnets:: Announce the full subnet in the EVPN network. |
343 | If you have silent VMs/CTs (for example, if you have multiple IPs and the | |
344 | anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be | |
345 | able to be reached inside the EVPN network). Optional. | |
4eb24270 | 346 | |
bcb72b25 SL |
347 | Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery) |
348 | packets. This is required if you use floating IPs in your VMs (IP and MAC | |
349 | addresses are being moved between systems). Optional. | |
f4e692a3 | 350 | |
bcb72b25 SL |
351 | Route-target Import:: Allows you to import a list of external EVPN route |
352 | targets. Used for cross-DC or different EVPN network interconnects. Optional. | |
4eb24270 | 353 | |
5899fa0e | 354 | MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes |
bcb72b25 SL |
355 | less than the maximal MTU of the outgoing physical interface. Optional, |
356 | defaults to 1450. | |
1556b768 AD |
357 | |
358 | ||
7823f52e | 359 | [[pvesdn_config_vnet]] |
a6af82c7 AD |
360 | VNets |
361 | ----- | |
362 | ||
bcb72b25 SL |
363 | After creating a virtual network (VNet) through the SDN GUI, a local network |
364 | interface with the same name is available on each node. To connect a guest to the | |
365 | VNet, assign the interface to the guest and set the IP address accordingly. | |
a6af82c7 | 366 | |
bcb72b25 SL |
367 | Depending on the zone, these options have different meanings and are explained |
368 | in the respective zone section in this document. | |
a6af82c7 | 369 | |
bcb72b25 SL |
370 | WARNING: In the current state, some options may have no effect or won't work in |
371 | certain zones. | |
a6af82c7 | 372 | |
bcb72b25 SL |
373 | VNet configuration options: |
374 | ||
375 | ID:: An up to 8 character ID to identify a VNet | |
376 | ||
377 | Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional | |
a6af82c7 AD |
378 | |
379 | Zone:: The associated zone for this VNet | |
380 | ||
5899fa0e | 381 | Tag:: The unique VLAN or VXLAN ID |
a6af82c7 | 382 | |
bcb72b25 SL |
383 | VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration |
384 | in the quest. | |
385 | ||
a6af82c7 AD |
386 | |
387 | [[pvesdn_config_subnet]] | |
5899fa0e | 388 | Subnets |
bcb72b25 | 389 | ------- |
3093e62d | 390 | |
bcb72b25 SL |
391 | A subnet define a specific IP range, described by the CIDR network address. |
392 | Each VNet, can have one or more subnets. | |
1556b768 | 393 | |
3093e62d | 394 | A subnet can be used to: |
a6af82c7 | 395 | |
5899fa0e DW |
396 | * Restrict the IP addresses you can define on a specific VNet |
397 | * Assign routes/gateways on a VNet in layer 3 zones | |
398 | * Enable SNAT on a VNet in layer 3 zones | |
399 | * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins | |
3093e62d | 400 | * DNS registration through DNS plugins |
a6af82c7 | 401 | |
5899fa0e | 402 | If an IPAM server is associated with the subnet zone, the subnet prefix will be |
3093e62d | 403 | automatically registered in the IPAM. |
a6af82c7 | 404 | |
bcb72b25 | 405 | Subnet configuration options: |
a6af82c7 | 406 | |
5899fa0e | 407 | ID:: A CIDR network address, for example 10.0.0.0/8 |
a6af82c7 | 408 | |
5899fa0e DW |
409 | Gateway:: The IP address of the network's default gateway. On layer 3 zones |
410 | (Simple/EVPN plugins), it will be deployed on the VNet. | |
3093e62d | 411 | |
bcb72b25 SL |
412 | SNAT:: Enable Source NAT which allows VMs from inside a |
413 | VNet to connect to the outside network by forwarding the packets to the nodes | |
414 | outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes. | |
415 | Optional. | |
416 | ||
417 | DNS Zone Prefix:: Add a prefix to the domain registration, like | |
418 | <hostname>.prefix.<domain> Optional. | |
a6af82c7 AD |
419 | |
420 | ||
a6af82c7 AD |
421 | [[pvesdn_config_controllers]] |
422 | Controllers | |
423 | ----------- | |
424 | ||
bcb72b25 | 425 | Some zones implement a separated control and data plane that require an external |
a823e409 | 426 | controller to manage the VNet's control plane. |
bcb72b25 SL |
427 | |
428 | Currently, only the `EVPN` zone requires an external controller. | |
429 | ||
8ac25ffe | 430 | |
4e652aba | 431 | [[pvesdn_controller_plugin_evpn]] |
ee6e18c4 TL |
432 | EVPN Controller |
433 | ~~~~~~~~~~~~~~~ | |
1556b768 | 434 | |
bcb72b25 SL |
435 | The `EVPN`, zone requires an external controller to manage the control plane. |
436 | The EVPN controller plugin configures the Free Range Routing (frr) router. | |
437 | ||
438 | To enable the EVPN controller, you need to install frr on every node that shall | |
439 | participate in the EVPN zone. | |
1556b768 AD |
440 | |
441 | ---- | |
a6af82c7 | 442 | apt install frr frr-pythontools |
1556b768 AD |
443 | ---- |
444 | ||
bcb72b25 | 445 | EVPN controller configuration options: |
1556b768 | 446 | |
bcb72b25 | 447 | ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN |
5899fa0e DW |
448 | number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up |
449 | breaking global routing by mistake. | |
1556b768 | 450 | |
bcb72b25 SL |
451 | Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be |
452 | external nodes or route reflector servers) | |
1556b768 | 453 | |
1556b768 | 454 | |
a6af82c7 AD |
455 | [[pvesdn_controller_plugin_BGP]] |
456 | BGP Controller | |
bcb72b25 | 457 | ~~~~~~~~~~~~~~ |
1556b768 | 458 | |
5899fa0e DW |
459 | The BGP controller is not used directly by a zone. |
460 | You can use it to configure FRR to manage BGP peers. | |
1556b768 | 461 | |
5899fa0e | 462 | For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP. |
065b2147 | 463 | It can also be used to export EVPN routes to an external BGP peer. |
4cf52a0c | 464 | |
065b2147 FG |
465 | NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP |
466 | controller. | |
1556b768 | 467 | |
bcb72b25 | 468 | BGP controller configuration options: |
1556b768 | 469 | |
bcb72b25 | 470 | Node:: The node of this BGP controller |
9a4f3c95 | 471 | |
bcb72b25 | 472 | ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN |
5899fa0e DW |
473 | number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise |
474 | you could break global routing by mistake. | |
a6af82c7 | 475 | |
bcb72b25 | 476 | Peer:: A list of peer IP addresses you want to communicate with using the |
5899fa0e | 477 | underlying BGP network. |
a6af82c7 | 478 | |
bcb72b25 | 479 | EBGP:: If your peer's remote-AS is different, this enables EBGP. |
a6af82c7 | 480 | |
bcb72b25 | 481 | Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network |
5899fa0e | 482 | (for multipath). |
a6af82c7 | 483 | |
5899fa0e DW |
484 | ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are |
485 | not directly connected or they use loopback. | |
a6af82c7 | 486 | |
5899fa0e | 487 | bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN. |
f4e692a3 | 488 | |
e5900013 AD |
489 | |
490 | [[pvesdn_controller_plugin_ISIS]] | |
491 | ISIS Controller | |
492 | ~~~~~~~~~~~~~~~ | |
493 | ||
494 | The ISIS controller is not used directly by a zone. | |
bcb72b25 | 495 | You can use it to configure FRR to export EVPN routes to an ISIS domain. |
e5900013 | 496 | |
bcb72b25 | 497 | ISIS controller configuration options: |
e5900013 | 498 | |
bcb72b25 | 499 | Node:: The node of this ISIS controller. |
e5900013 | 500 | |
bcb72b25 | 501 | Domain:: A unique ISIS domain. |
e5900013 | 502 | |
bcb72b25 | 503 | Network Entity Title:: A Unique ISIS network address that identifies this node. |
e5900013 | 504 | |
bcb72b25 | 505 | Interfaces:: A list of physical interface(s) used by ISIS. |
e5900013 | 506 | |
bcb72b25 | 507 | Loopback:: Use a loopback or dummy interface as the source of the EVPN network |
e5900013 AD |
508 | (for multipath). |
509 | ||
bcb72b25 | 510 | |
a6af82c7 | 511 | [[pvesdn_config_ipam]] |
bcb72b25 SL |
512 | IPAM |
513 | ---- | |
514 | ||
515 | IP Address Management (IPAM) tools manage the IP addresses of clients on the | |
516 | network. SDN in {pve} uses IPAM for example to find free IP addresses for new | |
517 | guests. | |
a6af82c7 | 518 | |
bcb72b25 | 519 | A single IPAM instance can be associated with one or more zones. |
a6af82c7 AD |
520 | |
521 | ||
522 | [[pvesdn_ipam_plugin_pveipam]] | |
bcb72b25 SL |
523 | PVE IPAM Plugin |
524 | ~~~~~~~~~~~~~~~ | |
a6af82c7 | 525 | |
bcb72b25 | 526 | The default built-in IPAM for your {pve} cluster. |
a6af82c7 | 527 | |
a7ec9f05 SH |
528 | You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in |
529 | the SDN section of the datacenter configuration. This UI can be used to create, | |
530 | update and delete IP mappings. This is particularly convenient in conjunction | |
531 | with the xref:pvesdn_config_dhcp[DHCP feature]. | |
532 | ||
533 | If you are using DHCP, you can use the IPAM panel to create or edit leases for | |
534 | specific VMs, which enables you to change the IPs allocated via DHCP. When | |
535 | editing an IP of a VM that is using DHCP you must make sure to force the guest | |
536 | to acquire a new DHCP leases. This can usually be done by reloading the network | |
537 | stack of the guest or rebooting it. | |
a6af82c7 | 538 | |
bcb72b25 SL |
539 | [[pvesdn_ipam_plugin_netbox]] |
540 | NetBox IPAM Plugin | |
0f2a9dce | 541 | ~~~~~~~~~~~~~~~~~~ |
a6af82c7 | 542 | |
bcb72b25 SL |
543 | link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP |
544 | Address Management (IPAM) and datacenter infrastructure management (DCIM) tool. | |
a6af82c7 | 545 | |
bcb72b25 SL |
546 | To integrate NetBox with {pve} SDN, create an API token in NetBox as described |
547 | here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens | |
5899fa0e | 548 | |
bcb72b25 | 549 | The NetBox configuration properties are: |
5899fa0e | 550 | |
bcb72b25 | 551 | URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api` |
a6af82c7 | 552 | |
bcb72b25 | 553 | Token:: An API access token |
3093e62d | 554 | |
a6af82c7 | 555 | |
bcb72b25 SL |
556 | [[pvesdn_ipam_plugin_phpipam]] |
557 | phpIPAM Plugin | |
558 | ~~~~~~~~~~~~~~ | |
a6af82c7 | 559 | |
bcb72b25 SL |
560 | In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add |
561 | an API token with admin privileges to the application. | |
562 | ||
563 | The phpIPAM configuration properties are: | |
a6af82c7 | 564 | |
bcb72b25 SL |
565 | URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/` |
566 | ||
567 | Token:: An API access token | |
568 | ||
569 | Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default | |
570 | installations use `sectionid=1` for customers. | |
5899fa0e | 571 | |
a6af82c7 AD |
572 | |
573 | [[pvesdn_config_dns]] | |
3093e62d | 574 | DNS |
a6af82c7 | 575 | --- |
3093e62d TL |
576 | |
577 | The DNS plugin in {pve} SDN is used to define a DNS API server for registration | |
5899fa0e DW |
578 | of your hostname and IP address. A DNS configuration is associated with one or |
579 | more zones, to provide DNS registration for all the subnet IPs configured for | |
3093e62d | 580 | a zone. |
a6af82c7 AD |
581 | |
582 | [[pvesdn_dns_plugin_powerdns]] | |
5899fa0e | 583 | PowerDNS Plugin |
a6af82c7 AD |
584 | ~~~~~~~~~~~~~~~ |
585 | https://doc.powerdns.com/authoritative/http-api/index.html | |
586 | ||
5899fa0e | 587 | You need to enable the web server and the API in your PowerDNS config: |
1556b768 | 588 | |
40f4dfc2 | 589 | ---- |
a6af82c7 AD |
590 | api=yes |
591 | api-key=arandomgeneratedstring | |
592 | webserver=yes | |
593 | webserver-port=8081 | |
40f4dfc2 | 594 | ---- |
40f4dfc2 | 595 | |
5899fa0e | 596 | The PowerDNS configuration options are: |
a6af82c7 | 597 | |
3093e62d | 598 | url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost |
5899fa0e | 599 | |
3093e62d | 600 | key:: An API access key |
5899fa0e | 601 | |
3093e62d | 602 | ttl:: The default TTL for records |
1556b768 | 603 | |
1556b768 | 604 | |
a7ec9f05 SH |
605 | [[pvesdn_config_dhcp]] |
606 | DHCP | |
607 | ------ | |
608 | ||
609 | The DHCP plugin in {pve} SDN can be used to automatically deploy a DHCP server | |
610 | for a Zone. It provides DHCP for all Subnets in a Zone that have a DHCP range | |
611 | configured. Currently the only available backend plugin for DHCP is the dnsmasq | |
612 | plugin. | |
613 | ||
614 | The DHCP plugin works by allocating an IP in the IPAM plugin configured in the | |
615 | Zone when adding a new network interface to a VM/CT. You can find more | |
616 | information on how to configure an IPAM in the | |
617 | xref:pvesdn_config_ipam[respective section of our documentation]. | |
618 | ||
619 | When the VM starts, a mapping for the MAC address and IP gets created in the DHCP | |
620 | plugin of the zone. When the network interfaces is removed or the VM/CT are | |
621 | destroyed, then the entry in the IPAM and the DHCP server are deleted as well. | |
622 | ||
623 | NOTE: Some features (adding/editing/removing IP mappings) are currently only | |
624 | available when using the xref:pvesdn_ipam_plugin_pveipam[PVE IPAM plugin]. | |
625 | ||
626 | ||
627 | Configuration | |
628 | ~~~~~~~~~~~~~ | |
629 | ||
630 | You can enable automatic DHCP for a zone in the Web UI via the Zones panel and | |
631 | enabling DHCP in the advanced options of a zone. | |
632 | ||
633 | NOTE: Currently only Simple Zones have support for automatic DHCP | |
634 | ||
635 | After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be | |
636 | configured for the subnets in a Zone. In order to that, go to the Vnets panel and | |
637 | select the Subnet for which you want to configure DHCP ranges. In the edit | |
638 | dialogue you can configure DHCP ranges in the respective Tab. Alternatively you | |
639 | can set DHCP ranges for a Subnet via the following CLI command: | |
640 | ||
641 | ---- | |
642 | pvesh set /cluster/sdn/vnets/<vnet>/subnets/<subnet> | |
643 | -dhcp-range start-address=10.0.1.100,end-address=10.0.1.200 | |
644 | -dhcp-range start-address=10.0.2.100,end-address=10.0.2.200 | |
645 | ---- | |
646 | ||
647 | You also need to have a gateway configured for the subnet - otherwise | |
648 | automatic DHCP will not work. | |
649 | ||
650 | The DHCP plugin will then allocate IPs in the IPAM only in the configured | |
651 | ranges. | |
652 | ||
653 | Do not forget to follow the installation steps for the | |
3d5a1655 | 654 | xref:pvesdn_install_dhcp_ipam[dnsmasq DHCP plugin] as well. |
a7ec9f05 SH |
655 | |
656 | Plugins | |
657 | ~~~~~~~ | |
658 | ||
659 | Dnsmasq Plugin | |
660 | ^^^^^^^^^^^^^^ | |
661 | Currently this is the only DHCP plugin and therefore the plugin that gets used | |
662 | when you enable DHCP for a zone. | |
663 | ||
a7ec9f05 | 664 | .Installation |
3d5a1655 | 665 | For installation see the xref:pvesdn_install_dhcp_ipam[DHCP IPAM] section. |
a7ec9f05 SH |
666 | |
667 | .Configuration | |
668 | The plugin will create a new systemd service for each zone that dnsmasq gets | |
669 | deployed to. The name for the service is `dnsmasq@<zone>`. The lifecycle of this | |
670 | service is managed by the DHCP plugin. | |
671 | ||
672 | The plugin automatically generates the following configuration files in the | |
673 | folder `/etc/dnsmasq.d/<zone>`: | |
674 | ||
675 | `00-default.conf`:: | |
676 | This contains the default global configuration for a dnsmasq instance. | |
677 | ||
678 | `10-<zone>-<subnet_cidr>.conf`:: | |
679 | This file configures specific options for a subnet, such as the DNS server that | |
680 | should get configured via DHCP. | |
681 | ||
682 | `10-<zone>-<subnet_cidr>.ranges.conf`:: | |
683 | This file configures the DHCP ranges for the dnsmasq instance. | |
684 | ||
685 | `ethers`:: | |
686 | This file contains the MAC-address and IP mappings from the IPAM plugin. In | |
687 | order to override those mappings, please use the respective IPAM plugin rather | |
688 | than editing this file, as it will get overwritten by the dnsmasq plugin. | |
689 | ||
690 | You must not edit any of the above files, since they are managed by the DHCP | |
691 | plugin. In order to customize the dnsmasq configuration you can create | |
692 | additional files (e.g. `90-custom.conf`) in the configuration folder - they will | |
693 | not get changed by the dnsmasq DHCP plugin. | |
694 | ||
695 | Configuration files are read in order, so you can control the order of the | |
696 | configuration directives by naming your custom configuration files appropriately. | |
697 | ||
698 | DHCP leases are stored in the file `/var/lib/misc/dnsmasq.<zone>.leases`. | |
699 | ||
700 | When using the PVE IPAM plugin, you can update, create and delete DHCP leases. | |
701 | For more information please consult the documentation of | |
702 | xref:pvesdn_ipam_plugin_pveipam[the PVE IPAM plugin]. Changing DHCP leases is | |
703 | currently not supported for the other IPAM plugins. | |
704 | ||
bcb72b25 | 705 | [[pvesdn_setup_examples]] |
a6af82c7 AD |
706 | Examples |
707 | -------- | |
708 | ||
bcb72b25 SL |
709 | This section presents multiple configuration examples tailored for common SDN |
710 | use cases. It aims to offer tangible implementations, providing additional | |
711 | details to enhance comprehension of the available configuration options. | |
1556b768 | 712 | |
ee6e18c4 | 713 | |
bcb72b25 SL |
714 | [[pvesdn_setup_example_simple]] |
715 | Simple Zone Example | |
716 | ~~~~~~~~~~~~~~~~~~~ | |
1556b768 | 717 | |
bcb72b25 SL |
718 | Simple zone networks create an isolated network for quests on a single host to |
719 | connect to each other. | |
1556b768 | 720 | |
bcb72b25 SL |
721 | TIP: connection between quests are possible if all quests reside on a same host |
722 | but cannot be reached on other nodes. | |
1556b768 | 723 | |
bcb72b25 SL |
724 | * Create a simple zone named `simple`. |
725 | * Add a VNet names `vnet1`. | |
726 | * Create a Subnet with a gateway and the SNAT option enabled. | |
727 | * This creates a network bridge `vnet1` on the node. Assign this bridge to the | |
728 | quests that shall join the network and configure an IP address. | |
1556b768 | 729 | |
bcb72b25 SL |
730 | The network interface configuration in two VMs may look like this which allows |
731 | them to communicate via the 10.0.1.0/24 network. | |
1556b768 AD |
732 | |
733 | ---- | |
bcb72b25 SL |
734 | allow-hotplug ens19 |
735 | iface ens19 inet static | |
736 | address 10.0.1.14/24 | |
737 | ---- | |
1556b768 | 738 | |
bcb72b25 SL |
739 | ---- |
740 | allow-hotplug ens19 | |
741 | iface ens19 inet static | |
742 | address 10.0.1.15/24 | |
743 | ---- | |
1556b768 | 744 | |
bcb72b25 SL |
745 | |
746 | [[pvesdn_setup_example_nat]] | |
747 | Source NAT Example | |
748 | ~~~~~~~~~~~~~~~~~~ | |
749 | ||
750 | If you want to allow outgoing connections for quests in the simple network zone | |
751 | the simple zone offers a Source NAT (SNAT) option. | |
752 | ||
753 | Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a | |
754 | Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option. | |
755 | ||
756 | ---- | |
757 | Subnet: 172.16.0.0/24 | |
758 | Gateway: 172.16.0.1 | |
759 | SNAT: checked | |
1556b768 AD |
760 | ---- |
761 | ||
bcb72b25 SL |
762 | In the quests configure the static IP address inside the subnet's IP range. |
763 | ||
764 | The node itself will join this network with the Gateway IP '172.16.0.1' and | |
765 | function as the NAT gateway for quests within the subnet range. | |
766 | ||
767 | ||
768 | [[pvesdn_setup_example_vlan]] | |
769 | VLAN Setup Example | |
770 | ~~~~~~~~~~~~~~~~~~ | |
771 | ||
772 | When VMs on different nodes need to communicate through an isolated network, the | |
773 | VLAN zone allows network level isolation using VLAN tags. | |
774 | ||
775 | Create a VLAN zone named `myvlanzone`: | |
1556b768 AD |
776 | |
777 | ---- | |
bcb72b25 SL |
778 | ID: myvlanzone |
779 | Bridge: vmbr0 | |
1556b768 AD |
780 | ---- |
781 | ||
bcb72b25 SL |
782 | Create a VNet named `myvnet1` with VLAN tag 10 and the previously created |
783 | `myvlanzone`. | |
1556b768 AD |
784 | |
785 | ---- | |
bcb72b25 SL |
786 | ID: myvnet1 |
787 | Zone: myvlanzone | |
788 | Tag: 10 | |
1556b768 AD |
789 | ---- |
790 | ||
ee6e18c4 | 791 | Apply the configuration through the main SDN panel, to create VNets locally on |
5899fa0e | 792 | each node. |
1556b768 | 793 | |
bcb72b25 | 794 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`. |
1556b768 | 795 | |
ee6e18c4 | 796 | Use the following network configuration for this VM: |
1556b768 AD |
797 | |
798 | ---- | |
799 | auto eth0 | |
800 | iface eth0 inet static | |
ee6e18c4 | 801 | address 10.0.3.100/24 |
1556b768 AD |
802 | ---- |
803 | ||
bcb72b25 SL |
804 | Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet |
805 | `myvnet1` as vm1. | |
ee6e18c4 TL |
806 | |
807 | Use the following network configuration for this VM: | |
808 | ||
1556b768 AD |
809 | ---- |
810 | auto eth0 | |
811 | iface eth0 inet static | |
ee6e18c4 | 812 | address 10.0.3.101/24 |
1556b768 AD |
813 | ---- |
814 | ||
bcb72b25 | 815 | Following this, you should be able to ping between both VMs using that network. |
1556b768 AD |
816 | |
817 | ||
4e652aba TL |
818 | [[pvesdn_setup_example_qinq]] |
819 | QinQ Setup Example | |
a6af82c7 | 820 | ~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 821 | |
1556b768 | 822 | |
bcb72b25 SL |
823 | This example configures two QinQ zones and adds two VMs to each zone to |
824 | demonstrate the additional layer of VLAN tags which allows the configuration of | |
825 | more isolated VLANs. | |
1556b768 | 826 | |
bcb72b25 SL |
827 | A typical use case for this configuration is a hosting provider that provides an |
828 | isolated network to customers for VM communication but isolates the VMs from | |
829 | other customers. | |
1556b768 | 830 | |
bcb72b25 | 831 | Create a QinQ zone named `qinqzone1` with service VLAN 20 |
1556b768 AD |
832 | |
833 | ---- | |
bcb72b25 SL |
834 | ID: qinqzone1 |
835 | Bridge: vmbr0 | |
836 | Service VLAN: 20 | |
1556b768 AD |
837 | ---- |
838 | ||
bcb72b25 | 839 | Create another QinQ zone named `qinqzone2` with service VLAN 30 |
1556b768 | 840 | ---- |
bcb72b25 SL |
841 | ID: qinqzone2 |
842 | Bridge: vmbr0 | |
843 | Service VLAN: 30 | |
1556b768 AD |
844 | ---- |
845 | ||
bcb72b25 SL |
846 | Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created |
847 | `qinqzone1` zone. | |
1556b768 AD |
848 | |
849 | ---- | |
bcb72b25 SL |
850 | ID: qinqvnet1 |
851 | Zone: qinqzone1 | |
852 | Tag: 100 | |
1556b768 AD |
853 | ---- |
854 | ||
bcb72b25 | 855 | Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone. |
1556b768 AD |
856 | |
857 | ---- | |
bcb72b25 SL |
858 | ID: qinqvnet2 |
859 | Zone: qinqzone2 | |
860 | Tag: 100 | |
1556b768 AD |
861 | ---- |
862 | ||
e2b3622a | 863 | Apply the configuration on the main SDN web interface panel to create VNets |
bcb72b25 | 864 | locally on each node. |
1556b768 | 865 | |
bcb72b25 SL |
866 | Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network |
867 | interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge | |
868 | `qinqvnet2`. | |
ee6e18c4 | 869 | |
bcb72b25 SL |
870 | Inside the VM, configure the IP addresses of the interfaces, for example via |
871 | `/etc/network/interfaces`: | |
ee6e18c4 | 872 | |
1556b768 AD |
873 | ---- |
874 | auto eth0 | |
875 | iface eth0 inet static | |
bcb72b25 | 876 | address 10.0.3.101/24 |
1556b768 | 877 | ---- |
bcb72b25 SL |
878 | // TODO: systemd-network example |
879 | Configure all four VMs to have IP addresses from the '10.0.3.101' to | |
880 | '10.0.3.104' range. | |
1556b768 | 881 | |
bcb72b25 | 882 | Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as |
5899fa0e | 883 | between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs |
bcb72b25 | 884 | 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN. |
1556b768 | 885 | |
1556b768 | 886 | |
4e652aba | 887 | [[pvesdn_setup_example_vxlan]] |
ee6e18c4 | 888 | VXLAN Setup Example |
a6af82c7 | 889 | ~~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 890 | |
bcb72b25 SL |
891 | The example assumes a cluster with three nodes, with the node IP addresses |
892 | 192.168.0.1, 192.168.0.2 and 192.168.0.3. | |
1556b768 | 893 | |
bcb72b25 SL |
894 | Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the |
895 | peer address list. Use the default MTU of 1450 or configure accordingly. | |
1556b768 | 896 | |
1556b768 | 897 | ---- |
bcb72b25 SL |
898 | ID: myvxlanzone |
899 | Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3 | |
1556b768 AD |
900 | ---- |
901 | ||
bcb72b25 | 902 | Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created |
ee6e18c4 | 903 | previously. |
1556b768 AD |
904 | |
905 | ---- | |
bcb72b25 SL |
906 | ID: vxvnet1 |
907 | Zone: myvxlanzone | |
908 | Tag: 100000 | |
1556b768 AD |
909 | ---- |
910 | ||
e2b3622a | 911 | Apply the configuration on the main SDN web interface panel to create VNets |
ee6e18c4 | 912 | locally on each nodes. |
1556b768 | 913 | |
bcb72b25 | 914 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`. |
1556b768 | 915 | |
5899fa0e | 916 | Use the following network configuration for this VM (note the lower MTU). |
1556b768 AD |
917 | |
918 | ---- | |
919 | auto eth0 | |
920 | iface eth0 inet static | |
bcb72b25 SL |
921 | address 10.0.3.100/24 |
922 | mtu 1450 | |
1556b768 AD |
923 | ---- |
924 | ||
bcb72b25 SL |
925 | Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet |
926 | `vxvnet1` as vm1. | |
ee6e18c4 TL |
927 | |
928 | Use the following network configuration for this VM: | |
929 | ||
1556b768 AD |
930 | ---- |
931 | auto eth0 | |
932 | iface eth0 inet static | |
bcb72b25 SL |
933 | address 10.0.3.101/24 |
934 | mtu 1450 | |
1556b768 AD |
935 | ---- |
936 | ||
ee6e18c4 | 937 | Then, you should be able to ping between between 'vm1' and 'vm2'. |
1556b768 AD |
938 | |
939 | ||
4e652aba TL |
940 | [[pvesdn_setup_example_evpn]] |
941 | EVPN Setup Example | |
a6af82c7 | 942 | ~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 943 | |
bcb72b25 SL |
944 | The example assumes a cluster with three nodes (node1, node2, node3) with IP |
945 | addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3. | |
1556b768 | 946 | |
5899fa0e DW |
947 | Create an EVPN controller, using a private ASN number and the above node |
948 | addresses as peers. | |
1556b768 AD |
949 | |
950 | ---- | |
bcb72b25 SL |
951 | ID: myevpnctl |
952 | ASN#: 65000 | |
953 | Peers: 192.168.0.1,192.168.0.2,192.168.0.3 | |
1556b768 AD |
954 | ---- |
955 | ||
bcb72b25 SL |
956 | Create an EVPN zone named `myevpnzone`, assign the previously created |
957 | EVPN-controller and define 'node1' and 'node2' as exit nodes. | |
a6af82c7 | 958 | |
1556b768 | 959 | ---- |
bcb72b25 SL |
960 | ID: myevpnzone |
961 | VRF VXLAN Tag: 10000 | |
962 | Controller: myevpnctl | |
963 | MTU: 1450 | |
964 | VNet MAC Address: 32:F4:05:FE:6C:0A | |
965 | Exit Nodes: node1,node2 | |
1556b768 AD |
966 | ---- |
967 | ||
bcb72b25 SL |
968 | Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`. |
969 | ||
1556b768 | 970 | ---- |
bcb72b25 SL |
971 | ID: myvnet1 |
972 | Zone: myevpnzone | |
973 | Tag: 11000 | |
1556b768 AD |
974 | ---- |
975 | ||
bcb72b25 | 976 | Create a subnet on `myvnet1`: |
9f819242 | 977 | |
a6af82c7 | 978 | ---- |
bcb72b25 SL |
979 | Subnet: 10.0.1.0/24 |
980 | Gateway: 10.0.1.1 | |
a6af82c7 AD |
981 | ---- |
982 | ||
bcb72b25 | 983 | Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`. |
1556b768 AD |
984 | |
985 | ---- | |
bcb72b25 SL |
986 | ID: myvnet2 |
987 | Zone: myevpnzone | |
988 | Tag: 12000 | |
1556b768 AD |
989 | ---- |
990 | ||
bcb72b25 | 991 | Create a different subnet on `myvnet2``: |
9f819242 | 992 | |
a6af82c7 | 993 | ---- |
bcb72b25 SL |
994 | Subnet: 10.0.2.0/24 |
995 | Gateway: 10.0.2.1 | |
a6af82c7 AD |
996 | ---- |
997 | ||
e2b3622a | 998 | Apply the configuration from the main SDN web interface panel to create VNets |
bcb72b25 | 999 | locally on each node and generate the FRR configuration. |
1556b768 | 1000 | |
bcb72b25 | 1001 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`. |
1556b768 | 1002 | |
bcb72b25 | 1003 | Use the following network configuration for 'vm1': |
1556b768 AD |
1004 | |
1005 | ---- | |
1006 | auto eth0 | |
1007 | iface eth0 inet static | |
bcb72b25 SL |
1008 | address 10.0.1.100/24 |
1009 | gateway 10.0.1.1 | |
1010 | mtu 1450 | |
1556b768 AD |
1011 | ---- |
1012 | ||
bcb72b25 SL |
1013 | Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet |
1014 | `myvnet2`. | |
ee6e18c4 | 1015 | |
bcb72b25 | 1016 | Use the following network configuration for 'vm2': |
ee6e18c4 | 1017 | |
1556b768 AD |
1018 | ---- |
1019 | auto eth0 | |
1020 | iface eth0 inet static | |
bcb72b25 SL |
1021 | address 10.0.2.100/24 |
1022 | gateway 10.0.2.1 | |
1023 | mtu 1450 | |
1556b768 AD |
1024 | ---- |
1025 | ||
1026 | ||
bcb72b25 | 1027 | Now you should be able to ping vm2 from vm1, and vm1 from vm2. |
1556b768 | 1028 | |
bcb72b25 | 1029 | If you ping an external IP from 'vm2' on the non-gateway node3, the packet |
a6af82c7 | 1030 | will go to the configured 'myvnet2' gateway, then will be routed to the exit |
ee6e18c4 TL |
1031 | nodes ('node1' or 'node2') and from there it will leave those nodes over the |
1032 | default gateway configured on node1 or node2. | |
1556b768 | 1033 | |
5899fa0e DW |
1034 | NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24' |
1035 | networks to node1 and node2 on your external gateway, so that the public network | |
1036 | can reply back. | |
1556b768 | 1037 | |
ee6e18c4 TL |
1038 | If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 |
1039 | and 10.0.2.0/24 in this example), will be announced dynamically. | |
ce84ae7e AD |
1040 | |
1041 | ||
bcb72b25 | 1042 | [[pvesdn_notes]] |
ce84ae7e AD |
1043 | Notes |
1044 | ----- | |
1045 | ||
4368fce0 AD |
1046 | Multiple EVPN Exit Nodes |
1047 | ~~~~~~~~~~~~~~~~~~~~~~~~ | |
1048 | ||
5b105f3e TL |
1049 | If you have multiple gateway nodes, you should disable the `rp_filter` (Strict |
1050 | Reverse Path Filter) option, because packets can arrive at one node but go out | |
1051 | from another node. | |
4368fce0 | 1052 | |
bcb72b25 SL |
1053 | Add the following to `/etc/sysctl.conf`: |
1054 | ||
4368fce0 AD |
1055 | ----- |
1056 | net.ipv4.conf.default.rp_filter=0 | |
1057 | net.ipv4.conf.all.rp_filter=0 | |
1058 | ----- | |
1059 | ||
448c1d39 TL |
1060 | VXLAN IPSEC Encryption |
1061 | ~~~~~~~~~~~~~~~~~~~~~~ | |
5899fa0e | 1062 | |
bcb72b25 SL |
1063 | To add IPSEC encryption on top of a VXLAN, this example shows how to use |
1064 | `strongswan`. | |
1065 | ||
1066 | You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for | |
1067 | IPv6 to handle encryption. | |
ce84ae7e | 1068 | |
448c1d39 TL |
1069 | So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC) |
1070 | + 50 (VXLAN) == 1500). | |
ce84ae7e | 1071 | |
bcb72b25 SL |
1072 | Install strongswan on the host. |
1073 | ||
ce84ae7e | 1074 | ---- |
8bfa192d | 1075 | apt install strongswan |
ce84ae7e AD |
1076 | ---- |
1077 | ||
bcb72b25 | 1078 | Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from |
448c1d39 | 1079 | the VXLAN UDP port '4789'. |
ce84ae7e AD |
1080 | |
1081 | ---- | |
1082 | conn %default | |
448c1d39 | 1083 | ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW |
ce84ae7e | 1084 | esp=aes256-sha1! |
448c1d39 | 1085 | leftfirewall=yes # this is necessary when using Proxmox VE firewall rules |
ce84ae7e AD |
1086 | |
1087 | conn output | |
1088 | rightsubnet=%dynamic[udp/4789] | |
1089 | right=%any | |
1090 | type=transport | |
1091 | authby=psk | |
1092 | auto=route | |
1093 | ||
1094 | conn input | |
1095 | leftsubnet=%dynamic[udp/4789] | |
1096 | type=transport | |
1097 | authby=psk | |
1098 | auto=route | |
1099 | ---- | |
1100 | ||
bcb72b25 | 1101 | Generate a pre-shared key with: |
ce84ae7e AD |
1102 | |
1103 | ---- | |
1104 | openssl rand -base64 128 | |
1105 | ---- | |
1106 | ||
bcb72b25 | 1107 | and add the key to `/etc/ipsec.secrets`, so that the file contents looks like: |
ce84ae7e AD |
1108 | |
1109 | ---- | |
1110 | : PSK <generatedbase64key> | |
1111 | ---- | |
448c1d39 | 1112 | |
bcb72b25 | 1113 | Copy the PSK and the configuration to all nodes participating in the VXLAN network. |