pvesdn.adoc

   1 [[chapter_pvesdn]]
   2 Software Defined Network
   3 ========================
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
   9 virtual networks (vnets) at datacenter level.
  10
  11 WARNING: SDN is currently an **experimental feature** in {pve}. This
  12 Documentation for it is also still under development, ask on our
  13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
  14
  15
  16 [[pvesdn_installation]]
  17 Installation
  18 ------------
  19
  20 To enable the experimental SDN integration, you need to install
  21 "libpve-network-perl" package
  22
  23 ----
  24 apt install libpve-network-perl
  25 ----
  26
  27 You need to have `ifupdown2` package installed on each node to manage local
  28 configuration reloading without reboot:
  29
  30 ----
  31 apt install ifupdown2
  32 ----
  33
  34 Basic Overview
  35 --------------
  36
  37 The {pve} SDN allows separation and fine grained control of Virtual Guests
  38 networks, using flexible software controlled configurations.
  39
  40 Separation consists of zones, a zone is it's own virtual separated network area.
  41 A 'VNet' is a type of a virtual network connected to a zone. Depending on which
  42 type or plugin the zone uses it can behave differently and offer different
  43 features, advantages or disadvantages.
  44 Normally a 'VNet' shows up as a common Linux bridge with either a VLAN or
  45 'VXLAN' tag, but some can also use layer 3 routing for control.
  46 The 'VNets' are deployed locally on each node, after configuration was committed
  47 from the cluster wide datacenter SDN administration interface.
  48
  49
  50 Main configuration
  51 ------------------
  52
  53 The configuration is done at datacenter (cluster-wide) level, it will be saved
  54 in configuration files located in the shared configuration file system:
  55 `/etc/pve/sdn`
  56
  57 On the web-interface  SDN feature have 4 main sections for the configuration
  58
  59 * SDN: a overview of the SDN state
  60
  61 * Zones: Create and manage the virtual separated network Zones
  62
  63 * VNets: The per-node building block to provide a Zone for VMs
  64
  65 * Controller: For complex setups to control Layer 3 routing
  66
  67
  68 [[pvesdn_config_main_sdn]]
  69 SDN
  70 ~~~
  71
  72 This is the main status panel. Here you can see deployment status of zones on
  73 different nodes.
  74
  75 There is an 'Apply' button, to push and reload local configuration on all
  76 cluster nodes nodes.
  77
  78
  79 [[pvesdn_config_zone]]
  80 Zones
  81 ~~~~~
  82
  83 A zone will define a virtually separated network.
  84
  85 It can use different technologies for separation:
  86
  87 * VLAN: Virtual LANs are the classic method to sub-divide a LAN
  88
  89 * QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
  90
  91 * VXLAN: (layer2 vxlan)
  92
  93 * bgp-evpn: vxlan using layer3 border gateway protocol routing
  94
  95 You can restrict a zone to specific nodes.
  96
  97 It's also possible to add permissions on a zone, to restrict user to use only a
  98 specific zone and only the VNets in that zone
  99
 100 [[pvesdn_config_vnet]]
 101 VNets
 102 ~~~~~
 103
 104 A `VNet` is in its basic form just a Linux bridge that will be deployed locally
 105 on the node and used for Virtual Machine communication.
 106
 107 VNet properties are:
 108
 109 * ID: a 8 characters ID to name and identify a VNet
 110
 111 * Alias: Optional longer name, if the ID isn't enough
 112
 113 * Zone: The associated zone for this VNet
 114
 115 * Tag: The unique VLAN or VXLAN id
 116
 117 * IPv4: an anycast IPv4 address, it will be configured on the underlying bridge
 118   on each node part of the Zone. It's only useful for `bgp-evpn` routing.
 119
 120 * IPv6: an anycast IPv6 address, it will be configured on the underlying bridge
 121   on each node part of the Zone. It's only useful for `bgp-evpn` routing.
 122
 123
 124 [[pvesdn_config_controllers]]
 125 Controllers
 126 ~~~~~~~~~~~
 127
 128 Some zone types need an external controller to manage the VNet control-plane.
 129 Currently this is only required for the `bgp-evpn` zone plugin.
 130
 131
 132 [[pvesdn_zone_plugins]]
 133 Zones Plugins
 134 -------------
 135
 136 Common options
 137 ~~~~~~~~~~~~~~
 138
 139 nodes:: Deploy and allow to use a VNets configured for this Zone only on these
 140 nodes.
 141
 142 [[pvesdn_zone_plugin_vlan]]
 143 VLAN Zones
 144 ~~~~~~~~~~
 145
 146 This is the simplest plugin, it will reuse an existing local Linux or OVS
 147 bridge, and manage VLANs on it.
 148 The benefit of using SDN module, is that you can create different zones with
 149 specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
 150
 151 Specific `VLAN` configuration options:
 152
 153 bridge:: Reuse this local bridge or OVS switch, already
 154 configured on *each* local node.
 155
 156 [[pvesdn_zone_plugin_qinq]]
 157 QinQ Zones
 158 ~~~~~~~~~~
 159
 160 QinQ is stacked VLAN. The first VLAN tag defined for the zone
 161 (so called 'service-vlan'), and the second VLAN tag defined for the vnets
 162
 163 NOTE: Your physical network switches must support stacked VLANs!
 164
 165 Specific QinQ configuration options:
 166
 167 bridge:: A local VLAN-aware bridge already configured on each local node
 168
 169 service vlan:: The main VLAN tag of this zone
 170
 171 mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
 172 For example, you reduce the MTU to `1496` if you physical interface MTU is
 173 `1500`.
 174
 175 [[pvesdn_zone_plugin_vxlan]]
 176 VXLAN Zones
 177 ~~~~~~~~~~~
 178
 179 The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
 180 network (named underlay). It encapsulate layer 2 Ethernet frames within layer
 181 4 UDP datagrams, using `4789` as the default destination port. You can, for
 182 example, create a private IPv4 VXLAN network on top of public internet network
 183 nodes.
 184 This is a layer2 tunnel only, no routing between different VNets is possible.
 185
 186 Each VNet will have use specific VXLAN id from the range (1 - 16777215).
 187
 188 Specific EVPN configuration options:
 189
 190 peers address list:: A list of IPs from all nodes through which you want to
 191 communicate. Can also be external nodes.
 192
 193 mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
 194 lower than the outgoing physical interface.
 195
 196 [[pvesdn_zone_plugin_evpn]]
 197 EVPN Zones
 198 ~~~~~~~~~~
 199
 200 This is the most complex of all supported plugins.
 201
 202 BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
 203 have an anycast IP-address and or MAC-address. The bridge IP is the same on each
 204 node, with this a virtual guest can use that address as gateway.
 205
 206 Routing can work across VNets from different zones through a VRF (Virtual
 207 Routing and Forwarding) interface.
 208
 209 Specific EVPN configuration options:
 210
 211 VRF VXLAN Tag:: This is a vxlan-id used for routing interconnect between vnets,
 212 it must be different than VXLAN-id of VNets
 213
 214 controller:: an EVPN-controller need to be defined first (see controller
 215 plugins section)
 216
 217 mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
 218 lower than the outgoing physical interface.
 219
 220
 221 [[pvesdn_controller_plugins]]
 222 Controllers Plugins
 223 -------------------
 224
 225 For complex zones requiring a control plane.
 226
 227 [[pvesdn_controller_plugin_evpn]]
 228 EVPN Controller
 229 ~~~~~~~~~~~~~~~
 230
 231 For `BGP-EVPN`, we need a controller to manage the control plane.
 232 The currently supported software controller is the "frr" router.
 233 You may need to install it on each node where you want to deploy EVPN zones.
 234
 235 ----
 236 apt install frr
 237 ----
 238
 239 Configuration options:
 240
 241 asn:: A unique BGP ASN number. It's highly recommended to use private ASN
 242 number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
 243 breaking, or get broken, by global routing by mistake.
 244
 245 peers:: An ip list of all nodes where you want to communicate (could be also
 246 external nodes or route reflectors servers)
 247
 248 Additionally, if you want to route traffic from a SDN BGP-EVPN network to
 249 external world:
 250
 251 gateway-nodes:: The proxmox nodes from where the bgp-evpn traffic will exit to
 252 external through the nodes default gateway
 253
 254 gateway-external-peers::  If you want that gateway nodes don't use the default
 255 gateway, but, for example, sent traffic to external BGP routers, which handle
 256 (reverse) routing then dynamically you can use. For example
 257 `192.168.0.253,192.168.0.254'
 258
 259
 260 [[pvesdn_local_deployment_monitoring]]
 261 Local Deployment Monitoring
 262 ---------------------------
 263
 264 After applying the configuration through the main SDN web-interface panel,
 265 the local network configuration is generated locally on each node in
 266 `/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
 267
 268 You need to add
 269 ----
 270 source /etc/network/interfaces.d/*
 271 ----
 272 at the end of /etc/network/interfaces to have the sdn config included
 273
 274 You can monitor the status of local zones and vnets through the main tree.
 275
 276
 277 [[pvesdn_setup_example_vlan]]
 278 VLAN Setup Example
 279 ------------------
 280
 281 TIP: While we show plain configuration content here, almost everything should
 282 be configurable using the web-interface only.
 283
 284 Node1: /etc/network/interfaces
 285
 286 ----
 287 auto vmbr0
 288 iface vmbr0 inet manual
 289         bridge-ports eno1
 290         bridge-stp off
 291         bridge-fd 0
 292         bridge-vlan-aware yes
 293         bridge-vids 2-4094
 294
 295 #management ip on vlan100
 296 auto vmbr0.100
 297 iface vmbr0.100 inet static
 298         address 192.168.0.1/24
 299
 300 source /etc/network/interfaces.d/*
 301 ----
 302
 303 Node2: /etc/network/interfaces
 304
 305 ----
 306 auto vmbr0
 307 iface vmbr0 inet manual
 308         bridge-ports eno1
 309         bridge-stp off
 310         bridge-fd 0
 311         bridge-vlan-aware yes
 312         bridge-vids 2-4094
 313
 314 #management ip on vlan100
 315 auto vmbr0.100
 316 iface vmbr0.100 inet static
 317         address 192.168.0.2/24
 318
 319 source /etc/network/interfaces.d/*
 320 ----
 321
 322 Create a VLAN zone named `myvlanzone':
 323
 324 ----
 325 id: myvlanzone
 326 bridge: vmbr0
 327 ----
 328
 329 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
 330 `myvlanzone' as it's zone.
 331
 332 ----
 333 id: myvnet1
 334 zone: myvlanzone
 335 tag: 10
 336 ----
 337
 338 Apply the configuration through the main SDN panel, to create VNets locally on
 339 each nodes.
 340
 341 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 342
 343 Use the following network configuration for this VM:
 344
 345 ----
 346 auto eth0
 347 iface eth0 inet static
 348         address 10.0.3.100/24
 349 ----
 350
 351 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 352 `myvnet1' as vm1.
 353
 354 Use the following network configuration for this VM:
 355
 356 ----
 357 auto eth0
 358 iface eth0 inet static
 359         address 10.0.3.101/24
 360 ----
 361
 362 Then, you should be able to ping between both VMs over that network.
 363
 364
 365 [[pvesdn_setup_example_qinq]]
 366 QinQ Setup Example
 367 ------------------
 368
 369 TIP: While we show plain configuration content here, almost everything should
 370 be configurable using the web-interface only.
 371
 372 Node1: /etc/network/interfaces
 373
 374 ----
 375 auto vmbr0
 376 iface vmbr0 inet manual
 377         bridge-ports eno1
 378         bridge-stp off
 379         bridge-fd 0
 380         bridge-vlan-aware yes
 381         bridge-vids 2-4094
 382
 383 #management ip on vlan100
 384 auto vmbr0.100
 385 iface vmbr0.100 inet static
 386         address 192.168.0.1/24
 387
 388 source /etc/network/interfaces.d/*
 389 ----
 390
 391 Node2: /etc/network/interfaces
 392
 393 ----
 394 auto vmbr0
 395 iface vmbr0 inet manual
 396         bridge-ports eno1
 397         bridge-stp off
 398         bridge-fd 0
 399         bridge-vlan-aware yes
 400         bridge-vids 2-4094
 401
 402 #management ip on vlan100
 403 auto vmbr0.100
 404 iface vmbr0.100 inet static
 405         address 192.168.0.2/24
 406
 407 source /etc/network/interfaces.d/*
 408 ----
 409
 410 Create an QinQ zone named `qinqzone1' with service VLAN 20
 411
 412 ----
 413 id: qinqzone1
 414 bridge: vmbr0
 415 service vlan: 20
 416 ----
 417
 418 Create another QinQ zone named `qinqzone2' with service VLAN 30
 419
 420 ----
 421 id: qinqzone2
 422 bridge: vmbr0
 423 service vlan: 30
 424 ----
 425
 426 Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
 427 created `qinqzone1' zone.
 428
 429 ----
 430 id: myvnet1
 431 zone: qinqzone1
 432 tag: 100
 433 ----
 434
 435 Create a `myvnet2' with customer VLAN-id 100 on the previously created
 436 `qinqzone2' zone.
 437
 438 ----
 439 id: myvnet2
 440 zone: qinqzone1
 441 tag: 100
 442 ----
 443
 444 Apply the configuration on the main SDN web-interface panel to create VNets
 445 locally on each nodes.
 446
 447 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 448
 449 Use the following network configuration for this VM:
 450
 451 ----
 452 auto eth0
 453 iface eth0 inet static
 454         address 10.0.3.100/24
 455 ----
 456
 457 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 458 `myvnet1' as vm1.
 459
 460 Use the following network configuration for this VM:
 461
 462 ----
 463 auto eth0
 464 iface eth0 inet static
 465         address 10.0.3.101/24
 466 ----
 467
 468 Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
 469 `myvnet2'.
 470
 471 Use the following network configuration for this VM:
 472
 473 ----
 474 auto eth0
 475 iface eth0 inet static
 476         address 10.0.3.102/24
 477 ----
 478
 479 Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
 480 `myvnet2' as vm3.
 481
 482 Use the following network configuration for this VM:
 483
 484 ----
 485 auto eth0
 486 iface eth0 inet static
 487         address 10.0.3.103/24
 488 ----
 489
 490 Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
 491 between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
 492 or 'vm4', as they are on a different zone with different service-vlan.
 493
 494
 495 [[pvesdn_setup_example_vxlan]]
 496 VXLAN Setup Example
 497 -------------------
 498
 499 TIP: While we show plain configuration content here, almost everything should
 500 be configurable using the web-interface only.
 501
 502 node1: /etc/network/interfaces
 503
 504 ----
 505 auto vmbr0
 506 iface vmbr0 inet static
 507         address 192.168.0.1/24
 508         gateway 192.168.0.254
 509         bridge-ports eno1
 510         bridge-stp off
 511         bridge-fd 0
 512         mtu 1500
 513
 514 source /etc/network/interfaces.d/*
 515 ----
 516
 517 node2: /etc/network/interfaces
 518
 519 ----
 520 auto vmbr0
 521 iface vmbr0 inet static
 522         address 192.168.0.2/24
 523         gateway 192.168.0.254
 524         bridge-ports eno1
 525         bridge-stp off
 526         bridge-fd 0
 527         mtu 1500
 528
 529 source /etc/network/interfaces.d/*
 530 ----
 531
 532 node3: /etc/network/interfaces
 533
 534 ----
 535 auto vmbr0
 536 iface vmbr0 inet static
 537         address 192.168.0.3/24
 538         gateway 192.168.0.254
 539         bridge-ports eno1
 540         bridge-stp off
 541         bridge-fd 0
 542         mtu 1500
 543
 544 source /etc/network/interfaces.d/*
 545 ----
 546
 547 Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
 548 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
 549 the nodes as peer address list.
 550
 551 ----
 552 id: myvxlanzone
 553 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
 554 mtu: 1450
 555 ----
 556
 557 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
 558 previously.
 559
 560 ----
 561 id: myvnet1
 562 zone: myvxlanzone
 563 tag: 100000
 564 ----
 565
 566 Apply the configuration on the main SDN web-interface panel to create VNets
 567 locally on each nodes.
 568
 569 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 570
 571 Use the following network configuration for this VM, note the lower MTU here.
 572
 573 ----
 574 auto eth0
 575 iface eth0 inet static
 576         address 10.0.3.100/24
 577         mtu 1450
 578 ----
 579
 580 Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
 581 `myvnet1' as vm1.
 582
 583 Use the following network configuration for this VM:
 584
 585 ----
 586 auto eth0
 587 iface eth0 inet static
 588         address 10.0.3.101/24
 589         mtu 1450
 590 ----
 591
 592 Then, you should be able to ping between between 'vm1' and 'vm2'.
 593
 594
 595 [[pvesdn_setup_example_evpn]]
 596 EVPN Setup Example
 597 ------------------
 598
 599 node1: /etc/network/interfaces
 600
 601 ----
 602 auto vmbr0
 603 iface vmbr0 inet static
 604         address 192.168.0.1/24
 605         gateway 192.168.0.254
 606         bridge-ports eno1
 607         bridge-stp off
 608         bridge-fd 0
 609         mtu 1500
 610
 611 source /etc/network/interfaces.d/*
 612 ----
 613
 614 node2: /etc/network/interfaces
 615
 616 ----
 617 auto vmbr0
 618 iface vmbr0 inet static
 619         address 192.168.0.2/24
 620         gateway 192.168.0.254
 621         bridge-ports eno1
 622         bridge-stp off
 623         bridge-fd 0
 624         mtu 1500
 625
 626 source /etc/network/interfaces.d/*
 627 ----
 628
 629 node3: /etc/network/interfaces
 630
 631 ----
 632 auto vmbr0
 633 iface vmbr0 inet static
 634         address 192.168.0.3/24
 635         gateway 192.168.0.254
 636         bridge-ports eno1
 637         bridge-stp off
 638         bridge-fd 0
 639         mtu 1500
 640
 641 source /etc/network/interfaces.d/*
 642 ----
 643
 644 Create a EVPN controller, using a private ASN number and above node addreesses
 645 as peers. Define 'node1' and 'node2' as gateway nodes.
 646
 647 ----
 648 id: myevpnctl
 649 asn: 65000
 650 peers: 192.168.0.1,192.168.0.2,192.168.0.3
 651 gateway nodes: node1,node2
 652 ----
 653
 654 Create an EVPN zone named `myevpnzone' using the previously created
 655 EVPN-controller.
 656
 657 ----
 658 id: myevpnzone
 659 vrf vxlan tag: 10000
 660 controller: myevpnctl
 661 mtu: 1450
 662 ----
 663
 664 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone', a IPv4
 665 CIDR network and a random MAC address.
 666
 667 ----
 668 id: myvnet1
 669 zone: myevpnzone
 670 tag: 11000
 671 ipv4: 10.0.1.1/24
 672 mac address: 8C:73:B2:7B:F9:60 #random generate mac address
 673 ----
 674
 675 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
 676 different IPv4 CIDR network and a different random MAC address than  `myvnet1'.
 677
 678 ----
 679 id: myvnet2
 680 zone: myevpnzone
 681 tag: 12000
 682 ipv4: 10.0.2.1/24
 683 mac address: 8C:73:B2:7B:F9:61  #random mac, need to be different on each vnet
 684 ----
 685
 686 Apply the configuration on the main SDN web-interface panel to create VNets
 687 locally on each nodes and generate the FRR config.
 688
 689
 690 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 691
 692 Use the following network configuration for this VM:
 693
 694 ----
 695 auto eth0
 696 iface eth0 inet static
 697         address 10.0.1.100/24
 698         gateway 10.0.1.1   #this is the ip of the vnet1
 699         mtu 1450
 700 ----
 701
 702 Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
 703 `myvnet2'.
 704
 705 Use the following network configuration for this VM:
 706
 707 ----
 708 auto eth0
 709 iface eth0 inet static
 710         address 10.0.2.100/24
 711         gateway 10.0.2.1   #this is the ip of the vnet2
 712         mtu 1450
 713 ----
 714
 715
 716 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
 717
 718 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
 719 will go to the configured 'myvnet2' gateway, then will be routed to gateway
 720 nodes ('node1' or 'node2') and from there it will leave those nodes over the
 721 default gateway configured on node1 or node2.
 722
 723 NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
 724 '10.0.2.0/24' network to node1, node2 on your external gateway, so that the
 725 public network can reply back.
 726
 727 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 728 and 10.0.2.0/24 in this example), will be announced dynamically.