]>
Commit | Line | Data |
---|---|---|
1556b768 | 1 | [[chapter_pvesdn]] |
3c126a21 | 2 | Software-Defined Network |
1556b768 AD |
3 | ======================== |
4 | ifndef::manvolnum[] | |
5 | :pve-toplevel: | |
6 | endif::manvolnum[] | |
7 | ||
3c126a21 | 8 | The **S**oftware-**D**efined **N**etwork (SDN) feature allows you to create |
5899fa0e | 9 | virtual networks (VNets) at the datacenter level. |
1556b768 | 10 | |
ee6e18c4 | 11 | WARNING: SDN is currently an **experimental feature** in {pve}. This |
5899fa0e | 12 | documentation for it is also still under development. Ask on our |
ee6e18c4 TL |
13 | xref:getting_help[mailing lists or in the forum] for questions and feedback. |
14 | ||
15 | ||
4e652aba | 16 | [[pvesdn_installation]] |
ee6e18c4 TL |
17 | Installation |
18 | ------------ | |
19 | ||
3c126a21 | 20 | To enable the experimental Software-Defined Network (SDN) integration, you need |
bcb72b25 | 21 | to install the `libpve-network-perl` package on every node: |
1556b768 AD |
22 | |
23 | ---- | |
684db7e3 | 24 | apt update |
bcb72b25 | 25 | apt install libpve-network-perl |
1556b768 AD |
26 | ---- |
27 | ||
bcb72b25 SL |
28 | NOTE: {pve} version 7 and above have the `ifupdown2` package installed by |
29 | default. If you originally installed your system with an older version, you need | |
30 | to explicitly install the `ifupdown2` package. | |
5899fa0e | 31 | |
bcb72b25 | 32 | After installation, you need to add the following line to the end of the |
5899fa0e DW |
33 | `/etc/network/interfaces` configuration file, so that the SDN configuration gets |
34 | included and activated. | |
1556b768 | 35 | |
a6af82c7 AD |
36 | ---- |
37 | source /etc/network/interfaces.d/* | |
38 | ---- | |
a6af82c7 AD |
39 | |
40 | ||
bcb72b25 SL |
41 | [[pvesdn_overview]] |
42 | Overview | |
43 | -------- | |
ee6e18c4 | 44 | |
5899fa0e DW |
45 | The {pve} SDN allows for separation and fine-grained control of virtual guest |
46 | networks, using flexible, software-controlled configurations. | |
ee6e18c4 | 47 | |
bcb72b25 SL |
48 | Separation is managed through *zones*, virtual networks (*VNets*), and |
49 | *subnets*. A zone is its own virtually separated network area. A VNet is a | |
50 | virtual network that belongs to a zone. A subnet is an IP range inside a VNet. | |
51 | ||
52 | Depending on the type of the zone, the network behaves differently and offers | |
53 | specific features, advantages, and limitations. | |
54 | ||
55 | Use cases for SDN range from an isolated private network on each individual node | |
56 | to complex overlay networks across multiple PVE clusters on different locations. | |
ee6e18c4 | 57 | |
bcb72b25 SL |
58 | After configuring an VNet in the cluster-wide datacenter SDN administration |
59 | interface, it is available as a common Linux bridge, locally on each node, to be | |
60 | assigned to VMs and Containers. | |
ee6e18c4 | 61 | |
bcb72b25 SL |
62 | |
63 | [[pvesdn_main_configuration]] | |
5899fa0e | 64 | Main Configuration |
a6af82c7 | 65 | ~~~~~~~~~~~~~~~~~~ |
1556b768 | 66 | |
bcb72b25 SL |
67 | Configuration is done at the web UI at datacenter level and is saved in files |
68 | located in the shared configuration file system at `/etc/pve/sdn`. | |
1556b768 | 69 | |
bcb72b25 | 70 | On the web interface, SDN features the following sections: |
1556b768 | 71 | |
bcb72b25 SL |
72 | * SDN:: Here you get an overview of the current active SDN state, and you can |
73 | apply all pending changes to the whole cluster. | |
1556b768 | 74 | |
bcb72b25 SL |
75 | * xref:pvesdn_config_zone[Zones]: Create and manage the virtually separated |
76 | network zones | |
a6af82c7 | 77 | |
bcb72b25 SL |
78 | * xref:pvesdn_config_vnets[VNets] VNets: Create virtual network bridges and |
79 | manage subnets | |
1556b768 | 80 | |
bcb72b25 SL |
81 | The Options category allows adding and managing additional services to be used |
82 | in your SDN setup. | |
1556b768 | 83 | |
bcb72b25 SL |
84 | * xref:pvesdn_config_controllers[Controllers]: For controlling layer 3 routing |
85 | in complex setups | |
a6af82c7 | 86 | |
bcb72b25 SL |
87 | * xref:pvesdn_config_ipam[IPAM]: Enables external for IP address management for |
88 | guests | |
a6af82c7 | 89 | |
bcb72b25 SL |
90 | * xref:pvesdn_config_dns[DNS]: Define a DNS server integration for registering |
91 | virtual guests' hostname and IP | |
5899fa0e | 92 | addresses |
1556b768 | 93 | |
4e652aba | 94 | [[pvesdn_config_main_sdn]] |
1556b768 AD |
95 | SDN |
96 | ~~~ | |
97 | ||
5899fa0e DW |
98 | This is the main status panel. Here you can see the deployment status of zones |
99 | on different nodes. | |
1556b768 | 100 | |
bcb72b25 | 101 | Pressing the 'Apply' button reloads the local configuration on all cluster |
5899fa0e | 102 | nodes. |
1556b768 | 103 | |
4e652aba | 104 | [[pvesdn_config_zone]] |
1556b768 | 105 | Zones |
a6af82c7 | 106 | ----- |
1556b768 | 107 | |
bcb72b25 | 108 | A zone defines a virtually separated network. Zones are restricted to |
5899fa0e DW |
109 | specific nodes and assigned permissions, in order to restrict users to a certain |
110 | zone and its contained VNets. | |
1556b768 | 111 | |
5899fa0e | 112 | Different technologies can be used for separation: |
1556b768 | 113 | |
bcb72b25 SL |
114 | * Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT) |
115 | ||
5899fa0e | 116 | * VLAN: Virtual LANs are the classic method of subdividing a LAN |
1556b768 | 117 | |
5899fa0e | 118 | * QinQ: Stacked VLAN (formally known as `IEEE 802.1ad`) |
1556b768 | 119 | |
bcb72b25 | 120 | * VXLAN: Layer 2 VXLAN network via a UDP tunnel |
a6af82c7 | 121 | |
bcb72b25 | 122 | * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing |
1556b768 | 123 | |
1556b768 | 124 | |
bcb72b25 SL |
125 | [[pvesdn_config_common_options]] |
126 | Common Options | |
a6af82c7 | 127 | ~~~~~~~~~~~~~~ |
1556b768 | 128 | |
5899fa0e | 129 | The following options are available for all zone types: |
3093e62d | 130 | |
bcb72b25 | 131 | Nodes:: The nodes which the zone and associated VNets should be deployed on. |
9e773815 | 132 | |
bcb72b25 SL |
133 | IPAM:: Use an IP Address Management (IPAM) tool to manage IPs in the |
134 | zone. Optional, defaults to `pve`. | |
1556b768 | 135 | |
bcb72b25 | 136 | DNS:: DNS API server. Optional. |
1556b768 | 137 | |
bcb72b25 | 138 | ReverseDNS:: Reverse DNS API server. Optional. |
1556b768 | 139 | |
bcb72b25 SL |
140 | DNSZone:: DNS domain name. Used to register hostnames, such as |
141 | `<hostname>.<domain>`. The DNS zone must already exist on the DNS server. Optional. | |
1556b768 AD |
142 | |
143 | ||
a6af82c7 AD |
144 | [[pvesdn_zone_plugin_simple]] |
145 | Simple Zones | |
146 | ~~~~~~~~~~~~ | |
1556b768 | 147 | |
bcb72b25 SL |
148 | This is the simplest plugin. It will create an isolated VNet bridge. This |
149 | bridge is not linked to a physical interface, and VM traffic is only local on | |
150 | each the node. | |
151 | It can be used in NAT or routed setups. | |
152 | ||
1556b768 | 153 | |
4e652aba | 154 | [[pvesdn_zone_plugin_vlan]] |
ee6e18c4 TL |
155 | VLAN Zones |
156 | ~~~~~~~~~~ | |
1556b768 | 157 | |
bcb72b25 SL |
158 | The VLAN plugin uses an existing local Linux or OVS bridge to connect to the |
159 | node's physical interface. It uses VLAN tagging defined in the VNet to isolate | |
160 | the network segments. This allows connectivity of VMs between different nodes. | |
161 | ||
162 | VLAN zone configuration options: | |
1556b768 | 163 | |
bcb72b25 SL |
164 | Bridge:: The local bridge or OVS switch, already configured on *each* node that |
165 | allows node-to-node connection. | |
1556b768 | 166 | |
1556b768 | 167 | |
4e652aba | 168 | [[pvesdn_zone_plugin_qinq]] |
ee6e18c4 TL |
169 | QinQ Zones |
170 | ~~~~~~~~~~ | |
1556b768 | 171 | |
bcb72b25 SL |
172 | QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for |
173 | isolation. The QinQ zone defines the outer VLAN tag (the 'Service VLAN') | |
174 | whereas the inner VLAN tag is defined by the VNet. | |
1556b768 | 175 | |
5899fa0e | 176 | NOTE: Your physical network switches must support stacked VLANs for this |
bcb72b25 | 177 | configuration. |
1556b768 | 178 | |
bcb72b25 | 179 | QinQ zone configuration options: |
1556b768 | 180 | |
bcb72b25 | 181 | Bridge:: A local, VLAN-aware bridge that is already configured on each local |
5899fa0e | 182 | node |
4e652aba | 183 | |
bcb72b25 | 184 | Service VLAN:: The main VLAN tag of this zone |
4e652aba | 185 | |
bcb72b25 | 186 | Service VLAN Protocol:: Allows you to choose between an 802.1q (default) or |
5899fa0e | 187 | 802.1ad service VLAN type. |
9a4f3c95 | 188 | |
bcb72b25 | 189 | MTU:: Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs. |
5899fa0e DW |
190 | For example, you must reduce the MTU to `1496` if you physical interface MTU is |
191 | `1500`. | |
1556b768 | 192 | |
bcb72b25 | 193 | |
4e652aba | 194 | [[pvesdn_zone_plugin_vxlan]] |
ee6e18c4 TL |
195 | VXLAN Zones |
196 | ~~~~~~~~~~~ | |
1556b768 | 197 | |
bcb72b25 SL |
198 | The VXLAN plugin establishes a tunnel (overlay) on top of an existing network |
199 | (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP | |
200 | datagrams using the default destination port `4789`. | |
201 | ||
202 | You have to configure the underlay network yourself to enable UDP connectivity | |
203 | between all peers. | |
1556b768 | 204 | |
bcb72b25 SL |
205 | You can, for example, create a VXLAN overlay network on top of public internet, |
206 | appearing to the VMs as if they share the same local Layer 2 network. | |
5899fa0e | 207 | |
bcb72b25 SL |
208 | WARNING: VXLAN on its own does does not provide any encryption. When joining |
209 | multiple sites via VXLAN, make sure to establish a secure connection between | |
210 | the site, for example by using a site-to-site VPN. | |
1556b768 | 211 | |
bcb72b25 | 212 | VXLAN zone configuration options: |
1556b768 | 213 | |
bcb72b25 SL |
214 | Peers Address List:: A list of IP addresses of each node in the VXLAN zone. This |
215 | can be external nodes reachable at this IP address. | |
216 | All nodes in the cluster need to be mentioned here. | |
4e652aba | 217 | |
bcb72b25 | 218 | MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes |
5899fa0e | 219 | lower than the outgoing physical interface. |
1556b768 | 220 | |
bcb72b25 | 221 | |
4e652aba | 222 | [[pvesdn_zone_plugin_evpn]] |
ee6e18c4 TL |
223 | EVPN Zones |
224 | ~~~~~~~~~~ | |
1556b768 | 225 | |
bcb72b25 SL |
226 | The EVPN zone creates a routable Layer 3 network, capable of spanning across |
227 | multiple clusters. This is achieved by establishing a VPN and utilizing BGP as | |
228 | the routing protocol. | |
1556b768 | 229 | |
bcb72b25 SL |
230 | The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge |
231 | IP is the same on each node, meaning a virtual guest can use this address as | |
232 | gateway. | |
1556b768 | 233 | |
ee6e18c4 TL |
234 | Routing can work across VNets from different zones through a VRF (Virtual |
235 | Routing and Forwarding) interface. | |
1556b768 | 236 | |
bcb72b25 | 237 | EVPN zone configuration options: |
1556b768 | 238 | |
bcb72b25 | 239 | VRF VXLAN ID:: A VXLAN-ID used for dedicated routing interconnect between VNets. |
5899fa0e | 240 | It must be different than the VXLAN-ID of the VNets. |
1556b768 | 241 | |
bcb72b25 | 242 | Controller:: The EVPN-controller to use for this zone. (See controller plugins |
5899fa0e | 243 | section). |
1556b768 | 244 | |
bcb72b25 SL |
245 | VNet MAC Address:: Anycast MAC address that gets assigned to all VNets in this |
246 | zone. Will be auto-generated if not defined. | |
a6af82c7 | 247 | |
bcb72b25 SL |
248 | Exit Nodes:: Nodes that shall be configured as exit gateways from the EVPN |
249 | network, through the real network. The configured nodes will announce a | |
250 | default route in the EVPN network. Optional. | |
a6af82c7 | 251 | |
bcb72b25 SL |
252 | Primary Exit Node:: If you use multiple exit nodes, force traffic through this |
253 | primary exit node, instead of load-balancing on all nodes. Optional but | |
254 | necessary if you want to use SNAT or if your upstream router doesn't support | |
5899fa0e | 255 | ECMP. |
f4e692a3 | 256 | |
bcb72b25 SL |
257 | Exit Nodes Local Routing:: This is a special option if you need to reach a VM/CT |
258 | service from an exit node. (By default, the exit nodes only allow forwarding | |
259 | traffic between real network and EVPN network). Optional. | |
f4e692a3 | 260 | |
bcb72b25 SL |
261 | Advertise Subnets:: Announce the full subnet in the EVPN network. |
262 | If you have silent VMs/CTs (for example, if you have multiple IPs and the | |
263 | anycast gateway doesn't see traffic from theses IPs, the IP addresses won't be | |
264 | able to be reached inside the EVPN network). Optional. | |
4eb24270 | 265 | |
bcb72b25 SL |
266 | Disable ARP ND Suppression:: Don't suppress ARP or ND (Neighbor Discovery) |
267 | packets. This is required if you use floating IPs in your VMs (IP and MAC | |
268 | addresses are being moved between systems). Optional. | |
f4e692a3 | 269 | |
bcb72b25 SL |
270 | Route-target Import:: Allows you to import a list of external EVPN route |
271 | targets. Used for cross-DC or different EVPN network interconnects. Optional. | |
4eb24270 | 272 | |
5899fa0e | 273 | MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes |
bcb72b25 SL |
274 | less than the maximal MTU of the outgoing physical interface. Optional, |
275 | defaults to 1450. | |
1556b768 AD |
276 | |
277 | ||
bcb72b25 | 278 | [[pvesdn_config_vnets]] |
a6af82c7 AD |
279 | VNets |
280 | ----- | |
281 | ||
bcb72b25 SL |
282 | After creating a virtual network (VNet) through the SDN GUI, a local network |
283 | interface with the same name is available on each node. To connect a guest to the | |
284 | VNet, assign the interface to the guest and set the IP address accordingly. | |
a6af82c7 | 285 | |
bcb72b25 SL |
286 | Depending on the zone, these options have different meanings and are explained |
287 | in the respective zone section in this document. | |
a6af82c7 | 288 | |
bcb72b25 SL |
289 | WARNING: In the current state, some options may have no effect or won't work in |
290 | certain zones. | |
a6af82c7 | 291 | |
bcb72b25 SL |
292 | VNet configuration options: |
293 | ||
294 | ID:: An up to 8 character ID to identify a VNet | |
295 | ||
296 | Comment:: More descriptive identifier. Assigned as an alias on the interface. Optional | |
a6af82c7 AD |
297 | |
298 | Zone:: The associated zone for this VNet | |
299 | ||
5899fa0e | 300 | Tag:: The unique VLAN or VXLAN ID |
a6af82c7 | 301 | |
bcb72b25 SL |
302 | VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration |
303 | in the quest. | |
304 | ||
a6af82c7 AD |
305 | |
306 | [[pvesdn_config_subnet]] | |
5899fa0e | 307 | Subnets |
bcb72b25 | 308 | ------- |
3093e62d | 309 | |
bcb72b25 SL |
310 | A subnet define a specific IP range, described by the CIDR network address. |
311 | Each VNet, can have one or more subnets. | |
1556b768 | 312 | |
3093e62d | 313 | A subnet can be used to: |
a6af82c7 | 314 | |
5899fa0e DW |
315 | * Restrict the IP addresses you can define on a specific VNet |
316 | * Assign routes/gateways on a VNet in layer 3 zones | |
317 | * Enable SNAT on a VNet in layer 3 zones | |
318 | * Auto assign IPs on virtual guests (VM or CT) through IPAM plugins | |
3093e62d | 319 | * DNS registration through DNS plugins |
a6af82c7 | 320 | |
5899fa0e | 321 | If an IPAM server is associated with the subnet zone, the subnet prefix will be |
3093e62d | 322 | automatically registered in the IPAM. |
a6af82c7 | 323 | |
bcb72b25 | 324 | Subnet configuration options: |
a6af82c7 | 325 | |
5899fa0e | 326 | ID:: A CIDR network address, for example 10.0.0.0/8 |
a6af82c7 | 327 | |
5899fa0e DW |
328 | Gateway:: The IP address of the network's default gateway. On layer 3 zones |
329 | (Simple/EVPN plugins), it will be deployed on the VNet. | |
3093e62d | 330 | |
bcb72b25 SL |
331 | SNAT:: Enable Source NAT which allows VMs from inside a |
332 | VNet to connect to the outside network by forwarding the packets to the nodes | |
333 | outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes. | |
334 | Optional. | |
335 | ||
336 | DNS Zone Prefix:: Add a prefix to the domain registration, like | |
337 | <hostname>.prefix.<domain> Optional. | |
a6af82c7 AD |
338 | |
339 | ||
a6af82c7 AD |
340 | [[pvesdn_config_controllers]] |
341 | Controllers | |
342 | ----------- | |
343 | ||
bcb72b25 SL |
344 | Some zones implement a separated control and data plane that require an external |
345 | external controller to manage the VNet's control plane. | |
346 | ||
347 | Currently, only the `EVPN` zone requires an external controller. | |
348 | ||
8ac25ffe | 349 | |
4e652aba | 350 | [[pvesdn_controller_plugin_evpn]] |
ee6e18c4 TL |
351 | EVPN Controller |
352 | ~~~~~~~~~~~~~~~ | |
1556b768 | 353 | |
bcb72b25 SL |
354 | The `EVPN`, zone requires an external controller to manage the control plane. |
355 | The EVPN controller plugin configures the Free Range Routing (frr) router. | |
356 | ||
357 | To enable the EVPN controller, you need to install frr on every node that shall | |
358 | participate in the EVPN zone. | |
1556b768 AD |
359 | |
360 | ---- | |
a6af82c7 | 361 | apt install frr frr-pythontools |
1556b768 AD |
362 | ---- |
363 | ||
bcb72b25 | 364 | EVPN controller configuration options: |
1556b768 | 365 | |
bcb72b25 | 366 | ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN |
5899fa0e DW |
367 | number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up |
368 | breaking global routing by mistake. | |
1556b768 | 369 | |
bcb72b25 SL |
370 | Peers:: An IP list of all nodes that are part of the EVPN zone. (could also be |
371 | external nodes or route reflector servers) | |
1556b768 | 372 | |
1556b768 | 373 | |
a6af82c7 AD |
374 | [[pvesdn_controller_plugin_BGP]] |
375 | BGP Controller | |
bcb72b25 | 376 | ~~~~~~~~~~~~~~ |
1556b768 | 377 | |
5899fa0e DW |
378 | The BGP controller is not used directly by a zone. |
379 | You can use it to configure FRR to manage BGP peers. | |
1556b768 | 380 | |
5899fa0e | 381 | For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP. |
065b2147 | 382 | It can also be used to export EVPN routes to an external BGP peer. |
4cf52a0c | 383 | |
065b2147 FG |
384 | NOTE: By default, for a simple full mesh EVPN, you don't need to define a BGP |
385 | controller. | |
1556b768 | 386 | |
bcb72b25 | 387 | BGP controller configuration options: |
1556b768 | 388 | |
bcb72b25 | 389 | Node:: The node of this BGP controller |
9a4f3c95 | 390 | |
bcb72b25 | 391 | ASN #:: A unique BGP ASN number. It's highly recommended to use a private ASN |
5899fa0e DW |
392 | number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise |
393 | you could break global routing by mistake. | |
a6af82c7 | 394 | |
bcb72b25 | 395 | Peer:: A list of peer IP addresses you want to communicate with using the |
5899fa0e | 396 | underlying BGP network. |
a6af82c7 | 397 | |
bcb72b25 | 398 | EBGP:: If your peer's remote-AS is different, this enables EBGP. |
a6af82c7 | 399 | |
bcb72b25 | 400 | Loopback Interface:: Use a loopback or dummy interface as the source of the EVPN network |
5899fa0e | 401 | (for multipath). |
a6af82c7 | 402 | |
5899fa0e DW |
403 | ebgp-mutltihop:: Increase the number of hops to reach peers, in case they are |
404 | not directly connected or they use loopback. | |
a6af82c7 | 405 | |
5899fa0e | 406 | bgp-multipath-as-path-relax:: Allow ECMP if your peers have different ASN. |
f4e692a3 | 407 | |
e5900013 AD |
408 | |
409 | [[pvesdn_controller_plugin_ISIS]] | |
410 | ISIS Controller | |
411 | ~~~~~~~~~~~~~~~ | |
412 | ||
413 | The ISIS controller is not used directly by a zone. | |
bcb72b25 | 414 | You can use it to configure FRR to export EVPN routes to an ISIS domain. |
e5900013 | 415 | |
bcb72b25 | 416 | ISIS controller configuration options: |
e5900013 | 417 | |
bcb72b25 | 418 | Node:: The node of this ISIS controller. |
e5900013 | 419 | |
bcb72b25 | 420 | Domain:: A unique ISIS domain. |
e5900013 | 421 | |
bcb72b25 | 422 | Network Entity Title:: A Unique ISIS network address that identifies this node. |
e5900013 | 423 | |
bcb72b25 | 424 | Interfaces:: A list of physical interface(s) used by ISIS. |
e5900013 | 425 | |
bcb72b25 | 426 | Loopback:: Use a loopback or dummy interface as the source of the EVPN network |
e5900013 AD |
427 | (for multipath). |
428 | ||
bcb72b25 | 429 | |
a6af82c7 | 430 | [[pvesdn_config_ipam]] |
bcb72b25 SL |
431 | IPAM |
432 | ---- | |
433 | ||
434 | IP Address Management (IPAM) tools manage the IP addresses of clients on the | |
435 | network. SDN in {pve} uses IPAM for example to find free IP addresses for new | |
436 | guests. | |
a6af82c7 | 437 | |
bcb72b25 | 438 | A single IPAM instance can be associated with one or more zones. |
a6af82c7 AD |
439 | |
440 | ||
441 | [[pvesdn_ipam_plugin_pveipam]] | |
bcb72b25 SL |
442 | PVE IPAM Plugin |
443 | ~~~~~~~~~~~~~~~ | |
a6af82c7 | 444 | |
bcb72b25 | 445 | The default built-in IPAM for your {pve} cluster. |
a6af82c7 | 446 | |
a6af82c7 | 447 | |
bcb72b25 SL |
448 | [[pvesdn_ipam_plugin_netbox]] |
449 | NetBox IPAM Plugin | |
450 | ~~~~~~~~~~~~~ | |
a6af82c7 | 451 | |
bcb72b25 SL |
452 | link:https://github.com/netbox-community/netbox[NetBox] is an open-source IP |
453 | Address Management (IPAM) and datacenter infrastructure management (DCIM) tool. | |
a6af82c7 | 454 | |
bcb72b25 SL |
455 | To integrate NetBox with {pve} SDN, create an API token in NetBox as described |
456 | here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens | |
5899fa0e | 457 | |
bcb72b25 | 458 | The NetBox configuration properties are: |
5899fa0e | 459 | |
bcb72b25 | 460 | URL:: The NetBox REST API endpoint: `http://yournetbox.domain.com/api` |
a6af82c7 | 461 | |
bcb72b25 | 462 | Token:: An API access token |
3093e62d | 463 | |
a6af82c7 | 464 | |
bcb72b25 SL |
465 | [[pvesdn_ipam_plugin_phpipam]] |
466 | phpIPAM Plugin | |
467 | ~~~~~~~~~~~~~~ | |
a6af82c7 | 468 | |
bcb72b25 SL |
469 | In link:https://phpipam.net/[phpIPAM] you need to create an "application" and add |
470 | an API token with admin privileges to the application. | |
471 | ||
472 | The phpIPAM configuration properties are: | |
a6af82c7 | 473 | |
bcb72b25 SL |
474 | URL:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/` |
475 | ||
476 | Token:: An API access token | |
477 | ||
478 | Section:: An integer ID. Sections are a group of subnets in phpIPAM. Default | |
479 | installations use `sectionid=1` for customers. | |
5899fa0e | 480 | |
a6af82c7 AD |
481 | |
482 | [[pvesdn_config_dns]] | |
3093e62d | 483 | DNS |
a6af82c7 | 484 | --- |
3093e62d TL |
485 | |
486 | The DNS plugin in {pve} SDN is used to define a DNS API server for registration | |
5899fa0e DW |
487 | of your hostname and IP address. A DNS configuration is associated with one or |
488 | more zones, to provide DNS registration for all the subnet IPs configured for | |
3093e62d | 489 | a zone. |
a6af82c7 AD |
490 | |
491 | [[pvesdn_dns_plugin_powerdns]] | |
5899fa0e | 492 | PowerDNS Plugin |
a6af82c7 AD |
493 | ~~~~~~~~~~~~~~~ |
494 | https://doc.powerdns.com/authoritative/http-api/index.html | |
495 | ||
5899fa0e | 496 | You need to enable the web server and the API in your PowerDNS config: |
1556b768 | 497 | |
40f4dfc2 | 498 | ---- |
a6af82c7 AD |
499 | api=yes |
500 | api-key=arandomgeneratedstring | |
501 | webserver=yes | |
502 | webserver-port=8081 | |
40f4dfc2 | 503 | ---- |
40f4dfc2 | 504 | |
5899fa0e | 505 | The PowerDNS configuration options are: |
a6af82c7 | 506 | |
3093e62d | 507 | url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost |
5899fa0e | 508 | |
3093e62d | 509 | key:: An API access key |
5899fa0e | 510 | |
3093e62d | 511 | ttl:: The default TTL for records |
1556b768 | 512 | |
1556b768 | 513 | |
bcb72b25 | 514 | [[pvesdn_setup_examples]] |
a6af82c7 AD |
515 | Examples |
516 | -------- | |
517 | ||
bcb72b25 SL |
518 | This section presents multiple configuration examples tailored for common SDN |
519 | use cases. It aims to offer tangible implementations, providing additional | |
520 | details to enhance comprehension of the available configuration options. | |
1556b768 | 521 | |
ee6e18c4 | 522 | |
bcb72b25 SL |
523 | [[pvesdn_setup_example_simple]] |
524 | Simple Zone Example | |
525 | ~~~~~~~~~~~~~~~~~~~ | |
1556b768 | 526 | |
bcb72b25 SL |
527 | Simple zone networks create an isolated network for quests on a single host to |
528 | connect to each other. | |
1556b768 | 529 | |
bcb72b25 SL |
530 | TIP: connection between quests are possible if all quests reside on a same host |
531 | but cannot be reached on other nodes. | |
1556b768 | 532 | |
bcb72b25 SL |
533 | * Create a simple zone named `simple`. |
534 | * Add a VNet names `vnet1`. | |
535 | * Create a Subnet with a gateway and the SNAT option enabled. | |
536 | * This creates a network bridge `vnet1` on the node. Assign this bridge to the | |
537 | quests that shall join the network and configure an IP address. | |
1556b768 | 538 | |
bcb72b25 SL |
539 | The network interface configuration in two VMs may look like this which allows |
540 | them to communicate via the 10.0.1.0/24 network. | |
1556b768 AD |
541 | |
542 | ---- | |
bcb72b25 SL |
543 | allow-hotplug ens19 |
544 | iface ens19 inet static | |
545 | address 10.0.1.14/24 | |
546 | ---- | |
1556b768 | 547 | |
bcb72b25 SL |
548 | ---- |
549 | allow-hotplug ens19 | |
550 | iface ens19 inet static | |
551 | address 10.0.1.15/24 | |
552 | ---- | |
1556b768 | 553 | |
bcb72b25 SL |
554 | |
555 | [[pvesdn_setup_example_nat]] | |
556 | Source NAT Example | |
557 | ~~~~~~~~~~~~~~~~~~ | |
558 | ||
559 | If you want to allow outgoing connections for quests in the simple network zone | |
560 | the simple zone offers a Source NAT (SNAT) option. | |
561 | ||
562 | Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a | |
563 | Subnet to the VNet `vnet1`, set a gateway IP and enable the SNAT option. | |
564 | ||
565 | ---- | |
566 | Subnet: 172.16.0.0/24 | |
567 | Gateway: 172.16.0.1 | |
568 | SNAT: checked | |
1556b768 AD |
569 | ---- |
570 | ||
bcb72b25 SL |
571 | In the quests configure the static IP address inside the subnet's IP range. |
572 | ||
573 | The node itself will join this network with the Gateway IP '172.16.0.1' and | |
574 | function as the NAT gateway for quests within the subnet range. | |
575 | ||
576 | ||
577 | [[pvesdn_setup_example_vlan]] | |
578 | VLAN Setup Example | |
579 | ~~~~~~~~~~~~~~~~~~ | |
580 | ||
581 | When VMs on different nodes need to communicate through an isolated network, the | |
582 | VLAN zone allows network level isolation using VLAN tags. | |
583 | ||
584 | Create a VLAN zone named `myvlanzone`: | |
1556b768 AD |
585 | |
586 | ---- | |
bcb72b25 SL |
587 | ID: myvlanzone |
588 | Bridge: vmbr0 | |
1556b768 AD |
589 | ---- |
590 | ||
bcb72b25 SL |
591 | Create a VNet named `myvnet1` with VLAN tag 10 and the previously created |
592 | `myvlanzone`. | |
1556b768 AD |
593 | |
594 | ---- | |
bcb72b25 SL |
595 | ID: myvnet1 |
596 | Zone: myvlanzone | |
597 | Tag: 10 | |
1556b768 AD |
598 | ---- |
599 | ||
ee6e18c4 | 600 | Apply the configuration through the main SDN panel, to create VNets locally on |
5899fa0e | 601 | each node. |
1556b768 | 602 | |
bcb72b25 | 603 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`. |
1556b768 | 604 | |
ee6e18c4 | 605 | Use the following network configuration for this VM: |
1556b768 AD |
606 | |
607 | ---- | |
608 | auto eth0 | |
609 | iface eth0 inet static | |
ee6e18c4 | 610 | address 10.0.3.100/24 |
1556b768 AD |
611 | ---- |
612 | ||
bcb72b25 SL |
613 | Create a second virtual machine ('vm2') on node2, with a vNIC on the same VNet |
614 | `myvnet1` as vm1. | |
ee6e18c4 TL |
615 | |
616 | Use the following network configuration for this VM: | |
617 | ||
1556b768 AD |
618 | ---- |
619 | auto eth0 | |
620 | iface eth0 inet static | |
ee6e18c4 | 621 | address 10.0.3.101/24 |
1556b768 AD |
622 | ---- |
623 | ||
bcb72b25 | 624 | Following this, you should be able to ping between both VMs using that network. |
1556b768 AD |
625 | |
626 | ||
4e652aba TL |
627 | [[pvesdn_setup_example_qinq]] |
628 | QinQ Setup Example | |
a6af82c7 | 629 | ~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 630 | |
1556b768 | 631 | |
bcb72b25 SL |
632 | This example configures two QinQ zones and adds two VMs to each zone to |
633 | demonstrate the additional layer of VLAN tags which allows the configuration of | |
634 | more isolated VLANs. | |
1556b768 | 635 | |
bcb72b25 SL |
636 | A typical use case for this configuration is a hosting provider that provides an |
637 | isolated network to customers for VM communication but isolates the VMs from | |
638 | other customers. | |
1556b768 | 639 | |
bcb72b25 | 640 | Create a QinQ zone named `qinqzone1` with service VLAN 20 |
1556b768 AD |
641 | |
642 | ---- | |
bcb72b25 SL |
643 | ID: qinqzone1 |
644 | Bridge: vmbr0 | |
645 | Service VLAN: 20 | |
1556b768 AD |
646 | ---- |
647 | ||
bcb72b25 | 648 | Create another QinQ zone named `qinqzone2` with service VLAN 30 |
1556b768 | 649 | ---- |
bcb72b25 SL |
650 | ID: qinqzone2 |
651 | Bridge: vmbr0 | |
652 | Service VLAN: 30 | |
1556b768 AD |
653 | ---- |
654 | ||
bcb72b25 SL |
655 | Create a VNet named `myvnet1` with VLAN-ID 100 on the previously created |
656 | `qinqzone1` zone. | |
1556b768 AD |
657 | |
658 | ---- | |
bcb72b25 SL |
659 | ID: qinqvnet1 |
660 | Zone: qinqzone1 | |
661 | Tag: 100 | |
1556b768 AD |
662 | ---- |
663 | ||
bcb72b25 | 664 | Create a `myvnet2` with VLAN-ID 100 on the `qinqzone2` zone. |
1556b768 AD |
665 | |
666 | ---- | |
bcb72b25 SL |
667 | ID: qinqvnet2 |
668 | Zone: qinqzone2 | |
669 | Tag: 100 | |
1556b768 AD |
670 | ---- |
671 | ||
ee6e18c4 | 672 | Apply the configuration on the main SDN web-interface panel to create VNets |
bcb72b25 | 673 | locally on each node. |
1556b768 | 674 | |
bcb72b25 SL |
675 | Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network |
676 | interfaces to vm1 and vm2 with bridge `qinqvnet1` and vm3 and vm4 with bridge | |
677 | `qinqvnet2`. | |
ee6e18c4 | 678 | |
bcb72b25 SL |
679 | Inside the VM, configure the IP addresses of the interfaces, for example via |
680 | `/etc/network/interfaces`: | |
ee6e18c4 | 681 | |
1556b768 AD |
682 | ---- |
683 | auto eth0 | |
684 | iface eth0 inet static | |
bcb72b25 | 685 | address 10.0.3.101/24 |
1556b768 | 686 | ---- |
bcb72b25 SL |
687 | // TODO: systemd-network example |
688 | Configure all four VMs to have IP addresses from the '10.0.3.101' to | |
689 | '10.0.3.104' range. | |
1556b768 | 690 | |
bcb72b25 | 691 | Now you should be able to ping between the VMs 'vm1' and 'vm2', as well as |
5899fa0e | 692 | between 'vm3' and 'vm4'. However, neither of VMs 'vm1' or 'vm2' can ping VMs |
bcb72b25 | 693 | 'vm3' or 'vm4', as they are on a different zone with a different service-VLAN. |
1556b768 | 694 | |
1556b768 | 695 | |
4e652aba | 696 | [[pvesdn_setup_example_vxlan]] |
ee6e18c4 | 697 | VXLAN Setup Example |
a6af82c7 | 698 | ~~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 699 | |
bcb72b25 SL |
700 | The example assumes a cluster with three nodes, with the node IP addresses |
701 | 192.168.0.1, 192.168.0.2 and 192.168.0.3. | |
1556b768 | 702 | |
bcb72b25 SL |
703 | Create a VXLAN zone named `myvxlanzone` and add all IPs from the nodes to the |
704 | peer address list. Use the default MTU of 1450 or configure accordingly. | |
1556b768 | 705 | |
1556b768 | 706 | ---- |
bcb72b25 SL |
707 | ID: myvxlanzone |
708 | Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3 | |
1556b768 AD |
709 | ---- |
710 | ||
bcb72b25 | 711 | Create a VNet named `vxvnet1` using the VXLAN zone `myvxlanzone` created |
ee6e18c4 | 712 | previously. |
1556b768 AD |
713 | |
714 | ---- | |
bcb72b25 SL |
715 | ID: vxvnet1 |
716 | Zone: myvxlanzone | |
717 | Tag: 100000 | |
1556b768 AD |
718 | ---- |
719 | ||
ee6e18c4 TL |
720 | Apply the configuration on the main SDN web-interface panel to create VNets |
721 | locally on each nodes. | |
1556b768 | 722 | |
bcb72b25 | 723 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `vxvnet1`. |
1556b768 | 724 | |
5899fa0e | 725 | Use the following network configuration for this VM (note the lower MTU). |
1556b768 AD |
726 | |
727 | ---- | |
728 | auto eth0 | |
729 | iface eth0 inet static | |
bcb72b25 SL |
730 | address 10.0.3.100/24 |
731 | mtu 1450 | |
1556b768 AD |
732 | ---- |
733 | ||
bcb72b25 SL |
734 | Create a second virtual machine ('vm2') on node3, with a vNIC on the same VNet |
735 | `vxvnet1` as vm1. | |
ee6e18c4 TL |
736 | |
737 | Use the following network configuration for this VM: | |
738 | ||
1556b768 AD |
739 | ---- |
740 | auto eth0 | |
741 | iface eth0 inet static | |
bcb72b25 SL |
742 | address 10.0.3.101/24 |
743 | mtu 1450 | |
1556b768 AD |
744 | ---- |
745 | ||
ee6e18c4 | 746 | Then, you should be able to ping between between 'vm1' and 'vm2'. |
1556b768 AD |
747 | |
748 | ||
4e652aba TL |
749 | [[pvesdn_setup_example_evpn]] |
750 | EVPN Setup Example | |
a6af82c7 | 751 | ~~~~~~~~~~~~~~~~~~ |
ee6e18c4 | 752 | |
bcb72b25 SL |
753 | The example assumes a cluster with three nodes (node1, node2, node3) with IP |
754 | addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3. | |
1556b768 | 755 | |
5899fa0e DW |
756 | Create an EVPN controller, using a private ASN number and the above node |
757 | addresses as peers. | |
1556b768 AD |
758 | |
759 | ---- | |
bcb72b25 SL |
760 | ID: myevpnctl |
761 | ASN#: 65000 | |
762 | Peers: 192.168.0.1,192.168.0.2,192.168.0.3 | |
1556b768 AD |
763 | ---- |
764 | ||
bcb72b25 SL |
765 | Create an EVPN zone named `myevpnzone`, assign the previously created |
766 | EVPN-controller and define 'node1' and 'node2' as exit nodes. | |
a6af82c7 | 767 | |
1556b768 | 768 | ---- |
bcb72b25 SL |
769 | ID: myevpnzone |
770 | VRF VXLAN Tag: 10000 | |
771 | Controller: myevpnctl | |
772 | MTU: 1450 | |
773 | VNet MAC Address: 32:F4:05:FE:6C:0A | |
774 | Exit Nodes: node1,node2 | |
1556b768 AD |
775 | ---- |
776 | ||
bcb72b25 SL |
777 | Create the first VNet named `myvnet1` using the EVPN zone `myevpnzone`. |
778 | ||
1556b768 | 779 | ---- |
bcb72b25 SL |
780 | ID: myvnet1 |
781 | Zone: myevpnzone | |
782 | Tag: 11000 | |
1556b768 AD |
783 | ---- |
784 | ||
bcb72b25 | 785 | Create a subnet on `myvnet1`: |
9f819242 | 786 | |
a6af82c7 | 787 | ---- |
bcb72b25 SL |
788 | Subnet: 10.0.1.0/24 |
789 | Gateway: 10.0.1.1 | |
a6af82c7 AD |
790 | ---- |
791 | ||
bcb72b25 | 792 | Create the second VNet named `myvnet2` using the same EVPN zone `myevpnzone`. |
1556b768 AD |
793 | |
794 | ---- | |
bcb72b25 SL |
795 | ID: myvnet2 |
796 | Zone: myevpnzone | |
797 | Tag: 12000 | |
1556b768 AD |
798 | ---- |
799 | ||
bcb72b25 | 800 | Create a different subnet on `myvnet2``: |
9f819242 | 801 | |
a6af82c7 | 802 | ---- |
bcb72b25 SL |
803 | Subnet: 10.0.2.0/24 |
804 | Gateway: 10.0.2.1 | |
a6af82c7 AD |
805 | ---- |
806 | ||
5899fa0e | 807 | Apply the configuration from the main SDN web-interface panel to create VNets |
bcb72b25 | 808 | locally on each node and generate the FRR configuration. |
1556b768 | 809 | |
bcb72b25 | 810 | Create a Debian-based virtual machine ('vm1') on node1, with a vNIC on `myvnet1`. |
1556b768 | 811 | |
bcb72b25 | 812 | Use the following network configuration for 'vm1': |
1556b768 AD |
813 | |
814 | ---- | |
815 | auto eth0 | |
816 | iface eth0 inet static | |
bcb72b25 SL |
817 | address 10.0.1.100/24 |
818 | gateway 10.0.1.1 | |
819 | mtu 1450 | |
1556b768 AD |
820 | ---- |
821 | ||
bcb72b25 SL |
822 | Create a second virtual machine ('vm2') on node2, with a vNIC on the other VNet |
823 | `myvnet2`. | |
ee6e18c4 | 824 | |
bcb72b25 | 825 | Use the following network configuration for 'vm2': |
ee6e18c4 | 826 | |
1556b768 AD |
827 | ---- |
828 | auto eth0 | |
829 | iface eth0 inet static | |
bcb72b25 SL |
830 | address 10.0.2.100/24 |
831 | gateway 10.0.2.1 | |
832 | mtu 1450 | |
1556b768 AD |
833 | ---- |
834 | ||
835 | ||
bcb72b25 | 836 | Now you should be able to ping vm2 from vm1, and vm1 from vm2. |
1556b768 | 837 | |
bcb72b25 | 838 | If you ping an external IP from 'vm2' on the non-gateway node3, the packet |
a6af82c7 | 839 | will go to the configured 'myvnet2' gateway, then will be routed to the exit |
ee6e18c4 TL |
840 | nodes ('node1' or 'node2') and from there it will leave those nodes over the |
841 | default gateway configured on node1 or node2. | |
1556b768 | 842 | |
5899fa0e DW |
843 | NOTE: You need to add reverse routes for the '10.0.1.0/24' and '10.0.2.0/24' |
844 | networks to node1 and node2 on your external gateway, so that the public network | |
845 | can reply back. | |
1556b768 | 846 | |
ee6e18c4 TL |
847 | If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 |
848 | and 10.0.2.0/24 in this example), will be announced dynamically. | |
ce84ae7e AD |
849 | |
850 | ||
bcb72b25 | 851 | [[pvesdn_notes]] |
ce84ae7e AD |
852 | Notes |
853 | ----- | |
854 | ||
4368fce0 AD |
855 | Multiple EVPN Exit Nodes |
856 | ~~~~~~~~~~~~~~~~~~~~~~~~ | |
857 | ||
5b105f3e TL |
858 | If you have multiple gateway nodes, you should disable the `rp_filter` (Strict |
859 | Reverse Path Filter) option, because packets can arrive at one node but go out | |
860 | from another node. | |
4368fce0 | 861 | |
bcb72b25 SL |
862 | Add the following to `/etc/sysctl.conf`: |
863 | ||
4368fce0 AD |
864 | ----- |
865 | net.ipv4.conf.default.rp_filter=0 | |
866 | net.ipv4.conf.all.rp_filter=0 | |
867 | ----- | |
868 | ||
448c1d39 TL |
869 | VXLAN IPSEC Encryption |
870 | ~~~~~~~~~~~~~~~~~~~~~~ | |
5899fa0e | 871 | |
bcb72b25 SL |
872 | To add IPSEC encryption on top of a VXLAN, this example shows how to use |
873 | `strongswan`. | |
874 | ||
875 | You`ll need to reduce the 'MTU' by additional 60 bytes for IPv4 or 80 bytes for | |
876 | IPv6 to handle encryption. | |
ce84ae7e | 877 | |
448c1d39 TL |
878 | So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC) |
879 | + 50 (VXLAN) == 1500). | |
ce84ae7e | 880 | |
bcb72b25 SL |
881 | Install strongswan on the host. |
882 | ||
ce84ae7e | 883 | ---- |
8bfa192d | 884 | apt install strongswan |
ce84ae7e AD |
885 | ---- |
886 | ||
bcb72b25 | 887 | Add configuration to `/etc/ipsec.conf`. We only need to encrypt traffic from |
448c1d39 | 888 | the VXLAN UDP port '4789'. |
ce84ae7e AD |
889 | |
890 | ---- | |
891 | conn %default | |
448c1d39 | 892 | ike=aes256-sha1-modp1024! # the fastest, but reasonably secure cipher on modern HW |
ce84ae7e | 893 | esp=aes256-sha1! |
448c1d39 | 894 | leftfirewall=yes # this is necessary when using Proxmox VE firewall rules |
ce84ae7e AD |
895 | |
896 | conn output | |
897 | rightsubnet=%dynamic[udp/4789] | |
898 | right=%any | |
899 | type=transport | |
900 | authby=psk | |
901 | auto=route | |
902 | ||
903 | conn input | |
904 | leftsubnet=%dynamic[udp/4789] | |
905 | type=transport | |
906 | authby=psk | |
907 | auto=route | |
908 | ---- | |
909 | ||
bcb72b25 | 910 | Generate a pre-shared key with: |
ce84ae7e AD |
911 | |
912 | ---- | |
913 | openssl rand -base64 128 | |
914 | ---- | |
915 | ||
bcb72b25 | 916 | and add the key to `/etc/ipsec.secrets`, so that the file contents looks like: |
ce84ae7e AD |
917 | |
918 | ---- | |
919 | : PSK <generatedbase64key> | |
920 | ---- | |
448c1d39 | 921 | |
bcb72b25 | 922 | Copy the PSK and the configuration to all nodes participating in the VXLAN network. |