pvesdn.adoc

   1 [[chapter_pvesdn]]
   2 Software Defined Network
   3 ========================
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
   9 virtual networks (vnets) at datacenter level.
  10
  11 WARNING: SDN is currently an **experimental feature** in {pve}. This
  12 Documentation for it is also still under development, ask on our
  13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
  14
  15
  16 [[pvesdn_installation]]
  17 Installation
  18 ------------
  19
  20 To enable the experimental SDN integration, you need to install
  21 "libpve-network-perl" package
  22
  23 ----
  24 apt install libpve-network-perl
  25 ----
  26
  27 You need to have `ifupdown2` package installed on each node to manage local
  28 configuration reloading without reboot:
  29
  30 ----
  31 apt install ifupdown2
  32 ----
  33
  34 Basic Overview
  35 --------------
  36
  37 The {pve} SDN allows separation and fine grained control of Virtual Guests
  38 networks, using flexible software controlled configurations.
  39
  40 Separation consists of zones, a zone is it's own virtual separated area.
  41 A Zone can be used by one or more 'VNets'. A 'VNet' is virtual network in a
  42 zone. Normally it shows up as a common Linux bridge with either a VLAN or
  43 'VXLAN' tag, or using layer 3 routing for control.
  44 The 'VNets' are deployed locally on each node, after configuration was commited
  45 from the cluster wide datacenter level.
  46
  47
  48 Main configuration
  49 ------------------
  50
  51 The configuration is done at datacenter (cluster-wide) level, it will be saved
  52 in configuration files located in the shared configuration file system:
  53 `/etc/pve/sdn`
  54
  55 On the web-interface  SDN feature have 4 main sections for the configuration
  56
  57 * SDN: a overview of the SDN state
  58
  59 * Zones: Create and manage the virtual separated network Zones
  60
  61 * VNets: The per-node building block to provide a Zone for VMs
  62
  63 * Controller:
  64
  65
  66 [[pvesdn_config_main_sdn]]
  67 SDN
  68 ~~~
  69
  70 This is the main status panel. Here you can see deployment status of zones on
  71 different nodes.
  72
  73 There is an 'Apply' button, to push and reload local configuration on all
  74 cluster nodes nodes.
  75
  76
  77 [[pvesdn_config_zone]]
  78 Zones
  79 ~~~~~
  80
  81 A zone will define a virtually separated network.
  82
  83 It can use different technologies for separation:
  84
  85 * VLAN: Virtual LANs are the classic method to sub-divide a LAN
  86
  87 * QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
  88
  89 * VXLAN: (layer2 vxlan)
  90
  91 * bgp-evpn: vxlan using layer3 border gateway protocol routing
  92
  93 You can restrict a zone to specific nodes.
  94
  95 It's also possible to add permissions on a zone, to restrict user to use only a
  96 specific zone and only the VNets in that zone
  97
  98 [[pvesdn_config_vnet]]
  99 VNets
 100 ~~~~~
 101
 102 A `VNet` is in its basic form just a Linux bridge that will be deployed locally
 103 on the node and used for Virtual Machine communication.
 104
 105 VNet properties are:
 106
 107 * ID: a 8 characters ID to name and identify a VNet
 108
 109 * Alias: Optional longer name, if the ID isn't enough
 110
 111 * Zone: The associated zone for this VNet
 112
 113 * Tag: The unique VLAN or VXLAN id
 114
 115 * IPv4: an anycast IPv4 address, it will be configured on the underlying bridge
 116   on each node part of the Zone. It's only useful for `bgp-evpn` routing.
 117
 118 * IPv6: an anycast IPv6 address, it will be configured on the underlying bridge
 119   on each node part of the Zone. It's only useful for `bgp-evpn` routing.
 120
 121
 122 [[pvesdn_config_controllers]]
 123 Controllers
 124 ~~~~~~~~~~~
 125
 126 Some zone types need an external controller to manage the VNet control-plane.
 127 Currently this is only required for the `bgp-evpn` zone plugin.
 128
 129
 130 [[pvesdn_zone_plugins]]
 131 Zones Plugins
 132 -------------
 133
 134 Common options
 135 ~~~~~~~~~~~~~~
 136
 137 nodes:: Deploy and allow to use a VNets configured for this Zone only on these
 138 nodes.
 139
 140 [[pvesdn_zone_plugin_vlan]]
 141 VLAN Zones
 142 ~~~~~~~~~~
 143
 144 This is the simplest plugin, it will reuse an existing local Linux or OVS
 145 bridge, and manage VLANs on it.
 146 The benefit of using SDN module, is that you can create different zones with
 147 specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
 148
 149 Specific `VLAN` configuration options:
 150
 151 bridge:: Reuse this local VLAN-aware bridge, or OVS interface, already
 152 configured on *each* local node.
 153
 154 [[pvesdn_zone_plugin_qinq]]
 155 QinQ Zones
 156 ~~~~~~~~~~
 157
 158 QinQ is stacked VLAN. The first VLAN tag defined for the zone
 159 (so called 'service-vlan'), and the second VLAN tag defined for the vnets
 160
 161 NOTE: Your physical network switchs must support stacked VLANs!
 162
 163 Specific QinQ configuration options:
 164
 165 bridge:: A local VLAN-aware bridge already configured on each local node
 166
 167 service vlan:: The main VLAN tag of this zone
 168
 169 mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
 170 For example, you reduce the MTU to `1496` if you physical interface MTU is
 171 `1500`.
 172
 173 [[pvesdn_zone_plugin_vxlan]]
 174 VXLAN Zones
 175 ~~~~~~~~~~~
 176
 177 The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
 178 network (named underlay). It encapsulate layer 2 Ethernet frames within layer
 179 4 UDP datagrams, using `4789` as the default destination port. You can, for
 180 example, create a private IPv4 VXLAN network on top of public internet network
 181 nodes.
 182 This is a layer2 tunnel only, no routing between different VNets is possible.
 183
 184 Each VNet will have use specific VXLAN id from the range (1 - 16777215).
 185
 186 Specific EVPN configuration options:
 187
 188 peers address list:: A list of IPs from all nodes through which you want to
 189 communicate. Can also be external nodes.
 190
 191 mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
 192 lower than the outgoing physical interface.
 193
 194 [[pvesdn_zone_plugin_evpn]]
 195 EVPN Zones
 196 ~~~~~~~~~~
 197
 198 This is the most complex of all supported plugins.
 199
 200 BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
 201 have an anycast IP-address and or MAC-address. The bridge IP is the same on each
 202 node, with this a virtual guest can use that address as gateway.
 203
 204 Routing can work across VNets from different zones through a VRF (Virtual
 205 Routing and Forwarding) interface.
 206
 207 Specific EVPN configuration options:
 208
 209 VRF VXLAN Tag:: This is a vxlan-id used for routing interconnect between vnets,
 210 it must be different than VXLAN-id of VNets
 211
 212 controller:: an EVPN-controller need to be defined first (see controller
 213 plugins section)
 214
 215 mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
 216 lower than the outgoing physical interface.
 217
 218
 219 [[pvesdn_controller_plugins]]
 220 Controllers Plugins
 221 -------------------
 222
 223 [[pvesdn_controller_plugin_evpn]]
 224 EVPN Controller
 225 ~~~~~~~~~~~~~~~
 226
 227 For `BGP-EVPN`, we need a controller to manage the control plane.
 228 The currently supported software controller is the "frr" router.
 229 You may need to install it on each node where you want to deploy EVPN zones.
 230
 231 ----
 232 apt install frr
 233 ----
 234
 235 Configuration options:
 236
 237 asn:: A unique BGP ASN number. It's highly recommended to use private ASN
 238 number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
 239 breaking, or get broken, by global routing by mistake.
 240
 241 peers:: An ip list of all nodes where you want to communicate (could be also
 242 external nodes or route reflectors servers)
 243
 244 Additionally, if you want to route traffic from a SDN BGP-EVPN network to
 245 external world:
 246
 247 gateway-nodes:: The proxmox nodes from where the bgp-evpn traffic will exit to
 248 external through the nodes default gateway
 249
 250 gateway-external-peers::  If you want that gateway nodes don't use the default
 251 gateway, but, for example, sent traffic to external BGP routers, which handle
 252 (reverse) routing then dynamically you can use. For example
 253 `192.168.0.253,192.168.0.254'
 254
 255
 256 [[pvesdn_local_deployment_monitoring]]
 257 Local Deployment Monitoring
 258 ---------------------------
 259
 260 After applying the configuration through the main SDN web-interface panel,
 261 the local network configuration is generated locally on each node in
 262 `/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
 263
 264 You can monitor the status of local zones and vnets through the main tree.
 265
 266
 267 [[pvesdn_setup_example_vlan]]
 268 VLAN Setup Example
 269 ------------------
 270
 271 TIP: While we show plain configuration content here, almost everything should
 272 be configurable using the web-interface only.
 273
 274 Node1: /etc/network/interfaces
 275
 276 ----
 277 auto vmbr0
 278 iface vmbr0 inet manual
 279         bridge-ports eno1
 280         bridge-stp off
 281         bridge-fd 0
 282         bridge-vlan-aware yes
 283         bridge-vids 2-4094
 284
 285 #management ip on vlan100
 286 auto vmbr0.100
 287 iface vmbr0.100 inet static
 288         address 192.168.0.1/24
 289
 290 source /etc/network/interfaces.d/*
 291 ----
 292
 293 Node2: /etc/network/interfaces
 294
 295 ----
 296 auto vmbr0
 297 iface vmbr0 inet manual
 298         bridge-ports eno1
 299         bridge-stp off
 300         bridge-fd 0
 301         bridge-vlan-aware yes
 302         bridge-vids 2-4094
 303
 304 #management ip on vlan100
 305 auto vmbr0.100
 306 iface vmbr0.100 inet static
 307         address 192.168.0.2/24
 308
 309 source /etc/network/interfaces.d/*
 310 ----
 311
 312 Create a VLAN zone named `myvlanzone':
 313
 314 ----
 315 id: myvlanzone
 316 bridge: vmbr0
 317 ----
 318
 319 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
 320 `myvlanzone' as it's zone.
 321
 322 ----
 323 id: myvnet1
 324 zone: myvlanzone
 325 tag: 10
 326 ----
 327
 328 Apply the configuration through the main SDN panel, to create VNets locally on
 329 each nodes.
 330
 331 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 332
 333 Use the following network configuration for this VM:
 334
 335 ----
 336 auto eth0
 337 iface eth0 inet static
 338         address 10.0.3.100/24
 339 ----
 340
 341 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 342 `myvnet1' as vm1.
 343
 344 Use the following network configuration for this VM:
 345
 346 ----
 347 auto eth0
 348 iface eth0 inet static
 349         address 10.0.3.101/24
 350 ----
 351
 352 Then, you should be able to ping between both VMs over that network.
 353
 354
 355 [[pvesdn_setup_example_qinq]]
 356 QinQ Setup Example
 357 ------------------
 358
 359 TIP: While we show plain configuration content here, almost everything should
 360 be configurable using the web-interface only.
 361
 362 Node1: /etc/network/interfaces
 363
 364 ----
 365 auto vmbr0
 366 iface vmbr0 inet manual
 367         bridge-ports eno1
 368         bridge-stp off
 369         bridge-fd 0
 370         bridge-vlan-aware yes
 371         bridge-vids 2-4094
 372
 373 #management ip on vlan100
 374 auto vmbr0.100
 375 iface vmbr0.100 inet static
 376         address 192.168.0.1/24
 377
 378 source /etc/network/interfaces.d/*
 379 ----
 380
 381 Node2: /etc/network/interfaces
 382
 383 ----
 384 auto vmbr0
 385 iface vmbr0 inet manual
 386         bridge-ports eno1
 387         bridge-stp off
 388         bridge-fd 0
 389         bridge-vlan-aware yes
 390         bridge-vids 2-4094
 391
 392 #management ip on vlan100
 393 auto vmbr0.100
 394 iface vmbr0.100 inet static
 395         address 192.168.0.2/24
 396
 397 source /etc/network/interfaces.d/*
 398 ----
 399
 400 Create an QinQ zone named `qinqzone1' with service VLAN 20
 401
 402 ----
 403 id: qinqzone1
 404 bridge: vmbr0
 405 service vlan: 20
 406 ----
 407
 408 Create another QinQ zone named `qinqzone2' with service VLAN 30
 409
 410 ----
 411 id: qinqzone2
 412 bridge: vmbr0
 413 service vlan: 30
 414 ----
 415
 416 Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
 417 created `qinqzone1' zone.
 418
 419 ----
 420 id: myvnet1
 421 zone: qinqzone1
 422 tag: 100
 423 ----
 424
 425 Create a `myvnet2' with customer VLAN-id 100 on the previously created
 426 `qinqzone2' zone.
 427
 428 ----
 429 id: myvnet2
 430 zone: qinqzone1
 431 tag: 100
 432 ----
 433
 434 Apply the configuration on the main SDN web-interface panel to create VNets
 435 locally on each nodes.
 436
 437 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 438
 439 Use the following network configuration for this VM:
 440
 441 ----
 442 auto eth0
 443 iface eth0 inet static
 444         address 10.0.3.100/24
 445 ----
 446
 447 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 448 `myvnet1' as vm1.
 449
 450 Use the following network configuration for this VM:
 451
 452 ----
 453 auto eth0
 454 iface eth0 inet static
 455         address 10.0.3.101/24
 456 ----
 457
 458 Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
 459 `myvnet2'.
 460
 461 Use the following network configuration for this VM:
 462
 463 ----
 464 auto eth0
 465 iface eth0 inet static
 466         address 10.0.3.102/24
 467 ----
 468
 469 Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
 470 `myvnet2' as vm3.
 471
 472 Use the following network configuration for this VM:
 473
 474 ----
 475 auto eth0
 476 iface eth0 inet static
 477         address 10.0.3.103/24
 478 ----
 479
 480 Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
 481 between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
 482 or 'vm4', as they are on a different zone with different service-vlan.
 483
 484
 485 [[pvesdn_setup_example_vxlan]]
 486 VXLAN Setup Example
 487 -------------------
 488
 489 TIP: While we show plain configuration content here, almost everything should
 490 be configurable using the web-interface only.
 491
 492 node1: /etc/network/interfaces
 493
 494 ----
 495 auto vmbr0
 496 iface vmbr0 inet static
 497         address 192.168.0.1/24
 498         gateway 192.168.0.254
 499         bridge-ports eno1
 500         bridge-stp off
 501         bridge-fd 0
 502         mtu 1500
 503
 504 source /etc/network/interfaces.d/*
 505 ----
 506
 507 node2: /etc/network/interfaces
 508
 509 ----
 510 auto vmbr0
 511 iface vmbr0 inet static
 512         address 192.168.0.2/24
 513         gateway 192.168.0.254
 514         bridge-ports eno1
 515         bridge-stp off
 516         bridge-fd 0
 517         mtu 1500
 518
 519 source /etc/network/interfaces.d/*
 520 ----
 521
 522 node3: /etc/network/interfaces
 523
 524 ----
 525 auto vmbr0
 526 iface vmbr0 inet static
 527         address 192.168.0.3/24
 528         gateway 192.168.0.254
 529         bridge-ports eno1
 530         bridge-stp off
 531         bridge-fd 0
 532         mtu 1500
 533
 534 source /etc/network/interfaces.d/*
 535 ----
 536
 537 Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
 538 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
 539 the nodes as peer address list.
 540
 541 ----
 542 id: myvxlanzone
 543 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
 544 mtu: 1450
 545 ----
 546
 547 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
 548 previously.
 549
 550 ----
 551 id: myvnet1
 552 zone: myvxlanzone
 553 tag: 100000
 554 ----
 555
 556 Apply the configuration on the main SDN web-interface panel to create VNets
 557 locally on each nodes.
 558
 559 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 560
 561 Use the following network configuration for this VM, note the lower MTU here.
 562
 563 ----
 564 auto eth0
 565 iface eth0 inet static
 566         address 10.0.3.100/24
 567         mtu 1450
 568 ----
 569
 570 Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
 571 `myvnet1' as vm1.
 572
 573 Use the following network configuration for this VM:
 574
 575 ----
 576 auto eth0
 577 iface eth0 inet static
 578         address 10.0.3.101/24
 579         mtu 1450
 580 ----
 581
 582 Then, you should be able to ping between between 'vm1' and 'vm2'.
 583
 584
 585 [[pvesdn_setup_example_evpn]]
 586 EVPN Setup Example
 587 ------------------
 588
 589 node1: /etc/network/interfaces
 590
 591 ----
 592 auto vmbr0
 593 iface vmbr0 inet static
 594         address 192.168.0.1/24
 595         gateway 192.168.0.254
 596         bridge-ports eno1
 597         bridge-stp off
 598         bridge-fd 0
 599         mtu 1500
 600
 601 source /etc/network/interfaces.d/*
 602 ----
 603
 604 node2: /etc/network/interfaces
 605
 606 ----
 607 auto vmbr0
 608 iface vmbr0 inet static
 609         address 192.168.0.2/24
 610         gateway 192.168.0.254
 611         bridge-ports eno1
 612         bridge-stp off
 613         bridge-fd 0
 614         mtu 1500
 615
 616 source /etc/network/interfaces.d/*
 617 ----
 618
 619 node3: /etc/network/interfaces
 620
 621 ----
 622 auto vmbr0
 623 iface vmbr0 inet static
 624         address 192.168.0.3/24
 625         gateway 192.168.0.254
 626         bridge-ports eno1
 627         bridge-stp off
 628         bridge-fd 0
 629         mtu 1500
 630
 631 source /etc/network/interfaces.d/*
 632 ----
 633
 634 Create a EVPN controller, using a private ASN number and above node addreesses
 635 as peers. Define 'node1' and 'node2' as gateway nodes.
 636
 637 ----
 638 id: myevpnctl
 639 asn: 65000
 640 peers: 192.168.0.1,192.168.0.2,192.168.0.3
 641 gateway nodes: node1,node2
 642 ----
 643
 644 Create an EVPN zone named `myevpnzone' using the previously created
 645 EVPN-controller.
 646
 647 ----
 648 id: myevpnzone
 649 vrf vxlan tag: 10000
 650 controller: myevpnctl
 651 mtu: 1450
 652 ----
 653
 654 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone', a IPv4
 655 CIDR network and a random MAC address.
 656
 657 ----
 658 id: myvnet1
 659 zone: myevpnzone
 660 tag: 11000
 661 ipv4: 10.0.1.1/24
 662 mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
 663 ----
 664
 665 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
 666 different IPv4 CIDR network and a different random MAC address than  `myvnet1'.
 667
 668 ----
 669 id: myvnet2
 670 zone: myevpnzone
 671 tag: 12000
 672 ipv4: 10.0.2.1/24
 673 mac address: 8C:73:B2:7B:F9:61  #random mac, need to be different on each vnet
 674 ----
 675
 676 Apply the configuration on the main SDN web-interface panel to create VNets
 677 locally on each nodes and generate the FRR config.
 678
 679
 680 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 681
 682 Use the following network configuration for this VM:
 683
 684 ----
 685 auto eth0
 686 iface eth0 inet static
 687         address 10.0.1.100/24
 688         gateway 10.0.1.1   #this is the ip of the vnet1
 689         mtu 1450
 690 ----
 691
 692 Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
 693 `myvnet2'.
 694
 695 Use the following network configuration for this VM:
 696
 697 ----
 698 auto eth0
 699 iface eth0 inet static
 700         address 10.0.2.100/24
 701         gateway 10.0.2.1   #this is the ip of the vnet2
 702         mtu 1450
 703 ----
 704
 705
 706 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
 707
 708 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
 709 will go to the configured 'myvnet2' gateway, then will be routed to gateway
 710 nodes ('node1' or 'node2') and from there it will leave those nodes over the
 711 default gateway configured on node1 or node2.
 712
 713 NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
 714 '10.0.2.0/24' network to node1, node2 on your external gateway, so that the
 715 public network can reply back.
 716
 717 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 718 and 10.0.2.0/24 in this example), will be announced dynamically.