pvesdn.adoc

   1 [[chapter_pvesdn]]
   2 Software Defined Network
   3 ========================
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
   9 virtual networks (vnets) at datacenter level.
  10
  11 WARNING: SDN is currently an **experimental feature** in {pve}. This
  12 Documentation for it is also still under development, ask on our
  13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
  14
  15
  16 Installation
  17 ------------
  18
  19 To enable the experimental SDN integration, you need to install
  20 "libpve-network-perl" package
  21
  22 ----
  23 apt install libpve-network-perl
  24 ----
  25
  26 You need to have `ifupdown2` package installed on each node to manage local
  27 configuration reloading without reboot:
  28
  29 ----
  30 apt install ifupdown2
  31 ----
  32
  33 Basic Overview
  34 --------------
  35
  36 The {pve} SDN allows separation and fine grained control of Virtual Guests
  37 networks, using flexible software controlled configurations.
  38
  39 Separation consists of zones, a zone is it's own virtual separated area.
  40 A Zone can be used by one or more 'VNets'. A 'VNet' is virtual network in a
  41 zone. Normally it shows up as a common Linux bridge with either a VLAN or
  42 'VXLAN' tag, or using layer 3 routing for control.
  43 The 'VNets' are deployed locally on each node, after configuration was commited
  44 from the cluster wide datacenter level.
  45
  46
  47 Main configuration
  48 ------------------
  49
  50 The configuration is done at datacenter (cluster-wide) level, it will be saved
  51 in configuration files located in the shared configuration file system:
  52 `/etc/pve/sdn`
  53
  54 On the web-interface  SDN feature have 4 main sections for the configuration
  55
  56 * SDN: a overview of the SDN state
  57
  58 * Zones: Create and manage the virtual separated network Zones
  59
  60 * VNets: The per-node building block to provide a Zone for VMs
  61
  62 * Controller:
  63
  64
  65 SDN
  66 ~~~
  67
  68 This is the main status panel. Here you can see deployment status of zones on
  69 different nodes.
  70
  71 There is an 'Apply' button, to push and reload local configuration on all
  72 cluster nodes nodes.
  73
  74
  75 Zones
  76 ~~~~~
  77
  78 A zone will define a virtually separated network.
  79
  80 It can use different technologies for separation:
  81
  82 * VLAN: Virtual LANs are the classic method to sub-divide a LAN
  83
  84 * QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
  85
  86 * VXLAN: (layer2 vxlan)
  87
  88 * bgp-evpn: vxlan using layer3 border gateway protocol routing
  89
  90 You can restrict a zone to specific nodes.
  91
  92 It's also possible to add permissions on a zone, to restrict user to use only a
  93 specific zone and only the VNets in that zone
  94
  95 VNets
  96 ~~~~~
  97
  98 A `VNet` is in its basic form just a Linux bridge that will be deployed locally
  99 on the node and used for Virtual Machine communication.
 100
 101 VNet properties are:
 102
 103 * ID: a 8 characters ID to name and identify a VNet
 104
 105 * Alias: Optional longer name, if the ID isn't enough
 106
 107 * Zone: The associated zone for this VNet
 108
 109 * Tag: The unique VLAN or VXLAN id
 110
 111 * IPv4: an anycast IPv4 address, it will be configured on the underlying bridge
 112   on each node part of the Zone. It's only useful for `bgp-evpn` routing.
 113
 114 * IPv6: an anycast IPv6 address, it will be configured on the underlying bridge
 115   on each node part of the Zone. It's only useful for `bgp-evpn` routing.
 116
 117
 118 Controllers
 119 ~~~~~~~~~~~
 120
 121 Some zone types (currently only the `bgp-evpn` plugin) need an external
 122 controller to manage the VNet control-plane.
 123
 124
 125 Zones Plugins
 126 -------------
 127
 128 Common options
 129 ~~~~~~~~~~~~~~
 130
 131 nodes:: deploy and allow to use a VNets configured for this Zone only on
 132 these nodes.
 133
 134
 135 VLAN Zones
 136 ~~~~~~~~~~
 137
 138 This is the simplest plugin, it will reuse an existing local Linux or OVS
 139 bridge, and manage VLANs on it.
 140 The benefit of using SDN module, is that you can create different zones with
 141 specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
 142
 143 Specific `VLAN` configuration options:
 144
 145 bridge:: Reuse this local VLAN-aware bridge, or OVS interface, already
 146 configured on *each* local node.
 147
 148 QinQ Zones
 149 ~~~~~~~~~~
 150
 151 QinQ is stacked VLAN. The first VLAN tag defined for the zone
 152 (so called 'service-vlan'), and the second VLAN tag defined for the vnets
 153
 154 NOTE: Your physical network switchs must support stacked VLANs!
 155
 156 Specific QinQ configuration options:
 157
 158 bridge:: a local VLAN-aware bridge already configured on each local node
 159 service vlan:: he main VLAN tag of this zone
 160 mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
 161 For example, you reduce the MTU to `1496` if you physical interface MTU is
 162 `1500`.
 163
 164 VXLAN Zones
 165 ~~~~~~~~~~~
 166
 167 The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
 168 network (named underlay). It encapsulate layer 2 Ethernet frames within layer
 169 4 UDP datagrams, using `4789` as the default destination port. You can, for
 170 example, create a private IPv4 VXLAN network on top of public internet network
 171 nodes.
 172 This is a layer2 tunnel only, no routing between different VNets is possible.
 173
 174 Each VNet will have use specific VXLAN id from the range (1 - 16777215).
 175
 176 Specific EVPN configuration options:
 177
 178 peers address list:: a list of IPs from all nodes where you want to communicate (can also be external nodes)
 179 mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes lower than the outgoing physical interface.
 180
 181 EVPN Zones
 182 ~~~~~~~~~~
 183
 184 This is the most complex of all supported plugins.
 185
 186 BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
 187 have an anycast IP-address and or MAC-address. The bridge IP is the same on each
 188 node, with this a virtual guest can use that address as gateway.
 189
 190 Routing can work across VNets from different zones through a VRF (Virtual
 191 Routing and Forwarding) interface.
 192
 193 Specific EVPN configuration options:
 194
 195 VRF VXLAN Tag:: This is a vxlan-id used for routing interconnect between vnets,
 196 it must be different than VXLAN-id of VNets
 197
 198 controller:: an EVPN-controller need to be defined first (see controller
 199 plugins section)
 200
 201 mtu:: because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
 202 lower than the outgoing physical interface.
 203
 204
 205 Controllers Plugins
 206 -------------------
 207
 208 EVPN Controller
 209 ~~~~~~~~~~~~~~~
 210
 211 For `BGP-EVPN`, we need a controller to manage the control plane.
 212 The currently supported software controller is the "frr" router.
 213 You may need to install it on each node where you want to deploy EVPN zones.
 214
 215 ----
 216 apt install frr
 217 ----
 218
 219 Configuration options:
 220
 221 asn:: a unique BGP ASN number. It's highly recommended to use private ASN
 222 number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
 223 breaking, or get broken, by global routing by mistake.
 224
 225 peers:: an ip list of all nodes where you want to communicate (could be also
 226 external nodes or route reflectors servers)
 227
 228 Additionally, if you want to route traffic from a SDN BGP-EVPN network to
 229 external world:
 230
 231 gateway-nodes:: The proxmox nodes from where the bgp-evpn traffic will exit to
 232 external through the nodes default gateway
 233
 234 If you want that gateway nodes don't use the default gateway, but, for example,
 235 sent traffic to external BGP routers
 236
 237 gateway-external-peers:: 192.168.0.253,192.168.0.254
 238
 239
 240 Local Deployment Monitoring
 241 ---------------------------
 242
 243 After applying the configuration through the main SDN web-interface panel,
 244 the local network configuration is generated locally on each node in
 245 `/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
 246
 247 You can monitor the status of local zones and vnets through the main tree.
 248
 249
 250 VLAN Setup Example
 251 ------------------
 252
 253 TIP: While we show plain configuration content here, almost everything should
 254 be configurable using the web-interface only.
 255
 256 Node1: /etc/network/interfaces
 257
 258 ----
 259 auto vmbr0
 260 iface vmbr0 inet manual
 261         bridge-ports eno1
 262         bridge-stp off
 263         bridge-fd 0
 264         bridge-vlan-aware yes
 265         bridge-vids 2-4094
 266
 267 #management ip on vlan100
 268 auto vmbr0.100
 269 iface vmbr0.100 inet static
 270         address 192.168.0.1/24
 271
 272 source /etc/network/interfaces.d/*
 273 ----
 274
 275 Node2: /etc/network/interfaces
 276
 277 ----
 278 auto vmbr0
 279 iface vmbr0 inet manual
 280         bridge-ports eno1
 281         bridge-stp off
 282         bridge-fd 0
 283         bridge-vlan-aware yes
 284         bridge-vids 2-4094
 285
 286 #management ip on vlan100
 287 auto vmbr0.100
 288 iface vmbr0.100 inet static
 289         address 192.168.0.2/24
 290
 291 source /etc/network/interfaces.d/*
 292 ----
 293
 294 Create a VLAN zone named `myvlanzone':
 295
 296 ----
 297 id: myvlanzone
 298 bridge: vmbr0
 299 ----
 300
 301 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
 302 `myvlanzone' as it's zone.
 303
 304 ----
 305 id: myvnet1
 306 zone: myvlanzone
 307 tag: 10
 308 ----
 309
 310 Apply the configuration through the main SDN panel, to create VNets locally on
 311 each nodes.
 312
 313 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 314
 315 Use the following network configuration for this VM:
 316
 317 ----
 318 auto eth0
 319 iface eth0 inet static
 320         address 10.0.3.100/24
 321 ----
 322
 323 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 324 `myvnet1' as vm1.
 325
 326 Use the following network configuration for this VM:
 327
 328 ----
 329 auto eth0
 330 iface eth0 inet static
 331         address 10.0.3.101/24
 332 ----
 333
 334 Then, you should be able to ping between both VMs over that network.
 335
 336
 337 QinQ setup example
 338 ------------------
 339
 340 TIP: While we show plain configuration content here, almost everything should
 341 be configurable using the web-interface only.
 342
 343 Node1: /etc/network/interfaces
 344
 345 ----
 346 auto vmbr0
 347 iface vmbr0 inet manual
 348         bridge-ports eno1
 349         bridge-stp off
 350         bridge-fd 0
 351         bridge-vlan-aware yes
 352         bridge-vids 2-4094
 353
 354 #management ip on vlan100
 355 auto vmbr0.100
 356 iface vmbr0.100 inet static
 357         address 192.168.0.1/24
 358
 359 source /etc/network/interfaces.d/*
 360 ----
 361
 362 Node2: /etc/network/interfaces
 363
 364 ----
 365 auto vmbr0
 366 iface vmbr0 inet manual
 367         bridge-ports eno1
 368         bridge-stp off
 369         bridge-fd 0
 370         bridge-vlan-aware yes
 371         bridge-vids 2-4094
 372
 373 #management ip on vlan100
 374 auto vmbr0.100
 375 iface vmbr0.100 inet static
 376         address 192.168.0.2/24
 377
 378 source /etc/network/interfaces.d/*
 379 ----
 380
 381 Create an QinQ zone named `qinqzone1' with service VLAN 20
 382
 383 ----
 384 id: qinqzone1
 385 bridge: vmbr0
 386 service vlan: 20
 387 ----
 388
 389 Create another QinQ zone named `qinqzone2' with service VLAN 30
 390
 391 ----
 392 id: qinqzone2
 393 bridge: vmbr0
 394 service vlan: 30
 395 ----
 396
 397 Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
 398 created `qinqzone1' zone.
 399
 400 ----
 401 id: myvnet1
 402 zone: qinqzone1
 403 tag: 100
 404 ----
 405
 406 Create a `myvnet2' with customer VLAN-id 100 on the previously created
 407 `qinqzone2' zone.
 408
 409 ----
 410 id: myvnet2
 411 zone: qinqzone1
 412 tag: 100
 413 ----
 414
 415 Apply the configuration on the main SDN web-interface panel to create VNets
 416 locally on each nodes.
 417
 418 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 419
 420 Use the following network configuration for this VM:
 421
 422 ----
 423 auto eth0
 424 iface eth0 inet static
 425         address 10.0.3.100/24
 426 ----
 427
 428 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 429 `myvnet1' as vm1.
 430
 431 Use the following network configuration for this VM:
 432
 433 ----
 434 auto eth0
 435 iface eth0 inet static
 436         address 10.0.3.101/24
 437 ----
 438
 439 Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
 440 `myvnet2'.
 441
 442 Use the following network configuration for this VM:
 443
 444 ----
 445 auto eth0
 446 iface eth0 inet static
 447         address 10.0.3.102/24
 448 ----
 449
 450 Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
 451 `myvnet2' as vm3.
 452
 453 Use the following network configuration for this VM:
 454
 455 ----
 456 auto eth0
 457 iface eth0 inet static
 458         address 10.0.3.103/24
 459 ----
 460
 461 Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
 462 between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
 463 or 'vm4', as they are on a different zone with different service-vlan.
 464
 465
 466 VXLAN Setup Example
 467 -------------------
 468
 469 node1: /etc/network/interfaces
 470
 471 ----
 472 auto vmbr0
 473 iface vmbr0 inet static
 474         address 192.168.0.1/24
 475         gateway 192.168.0.254
 476         bridge-ports eno1
 477         bridge-stp off
 478         bridge-fd 0
 479         mtu 1500
 480
 481 source /etc/network/interfaces.d/*
 482 ----
 483
 484 node2: /etc/network/interfaces
 485
 486 ----
 487 auto vmbr0
 488 iface vmbr0 inet static
 489         address 192.168.0.2/24
 490         gateway 192.168.0.254
 491         bridge-ports eno1
 492         bridge-stp off
 493         bridge-fd 0
 494         mtu 1500
 495
 496 source /etc/network/interfaces.d/*
 497 ----
 498
 499 node3: /etc/network/interfaces
 500
 501 ----
 502 auto vmbr0
 503 iface vmbr0 inet static
 504         address 192.168.0.3/24
 505         gateway 192.168.0.254
 506         bridge-ports eno1
 507         bridge-stp off
 508         bridge-fd 0
 509         mtu 1500
 510
 511 source /etc/network/interfaces.d/*
 512 ----
 513
 514 Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
 515 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
 516 the nodes as peer address list.
 517
 518 ----
 519 id: myvxlanzone
 520 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
 521 mtu: 1450
 522 ----
 523
 524 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
 525 previously.
 526
 527 ----
 528 id: myvnet1
 529 zone: myvxlanzone
 530 tag: 100000
 531 ----
 532
 533 Apply the configuration on the main SDN web-interface panel to create VNets
 534 locally on each nodes.
 535
 536 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 537
 538 Use the following network configuration for this VM, note the lower MTU here.
 539
 540 ----
 541 auto eth0
 542 iface eth0 inet static
 543         address 10.0.3.100/24
 544         mtu 1450
 545 ----
 546
 547 Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
 548 `myvnet1' as vm1.
 549
 550 Use the following network configuration for this VM:
 551
 552 ----
 553 auto eth0
 554 iface eth0 inet static
 555         address 10.0.3.101/24
 556         mtu 1450
 557 ----
 558
 559 Then, you should be able to ping between between 'vm1' and 'vm2'.
 560
 561
 562
 563 EVPN setup example
 564 ------------------
 565
 566 node1: /etc/network/interfaces
 567
 568 ----
 569 auto vmbr0
 570 iface vmbr0 inet static
 571         address 192.168.0.1/24
 572         gateway 192.168.0.254
 573         bridge-ports eno1
 574         bridge-stp off
 575         bridge-fd 0
 576         mtu 1500
 577
 578 source /etc/network/interfaces.d/*
 579 ----
 580
 581 node2: /etc/network/interfaces
 582
 583 ----
 584 auto vmbr0
 585 iface vmbr0 inet static
 586         address 192.168.0.2/24
 587         gateway 192.168.0.254
 588         bridge-ports eno1
 589         bridge-stp off
 590         bridge-fd 0
 591         mtu 1500
 592
 593 source /etc/network/interfaces.d/*
 594 ----
 595
 596 node3: /etc/network/interfaces
 597
 598 ----
 599 auto vmbr0
 600 iface vmbr0 inet static
 601         address 192.168.0.3/24
 602         gateway 192.168.0.254
 603         bridge-ports eno1
 604         bridge-stp off
 605         bridge-fd 0
 606         mtu 1500
 607
 608 source /etc/network/interfaces.d/*
 609 ----
 610
 611 Create a EVPN controller, using a private ASN number and above node addreesses
 612 as peers. Define 'node1' and 'node2' as gateway nodes.
 613
 614 ----
 615 id: myevpnctl
 616 asn: 65000
 617 peers: 192.168.0.1,192.168.0.2,192.168.0.3
 618 gateway nodes: node1,node2
 619 ----
 620
 621 Create an EVPN zone named `myevpnzone' using the previously created
 622 EVPN-controller.
 623
 624 ----
 625 id: myevpnzone
 626 vrf vxlan tag: 10000
 627 controller: myevpnctl
 628 mtu: 1450
 629 ----
 630
 631 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone', a IPv4
 632 CIDR network and a random MAC address.
 633
 634 ----
 635 id: myvnet1
 636 zone: myevpnzone
 637 tag: 11000
 638 ipv4: 10.0.1.1/24
 639 mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
 640 ----
 641
 642 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
 643 different IPv4 CIDR network and a different random MAC address than  `myvnet1'.
 644
 645 ----
 646 id: myvnet2
 647 zone: myevpnzone
 648 tag: 12000
 649 ipv4: 10.0.2.1/24
 650 mac address: 8C:73:B2:7B:F9:61  #random mac, need to be different on each vnet
 651 ----
 652
 653 Apply the configuration on the main SDN web-interface panel to create VNets
 654 locally on each nodes and generate the FRR config.
 655
 656
 657 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 658
 659 Use the following network configuration for this VM:
 660
 661 ----
 662 auto eth0
 663 iface eth0 inet static
 664         address 10.0.1.100/24
 665         gateway 10.0.1.1   #this is the ip of the vnet1
 666         mtu 1450
 667 ----
 668
 669 Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
 670 `myvnet2'.
 671
 672 Use the following network configuration for this VM:
 673
 674 ----
 675 auto eth0
 676 iface eth0 inet static
 677         address 10.0.2.100/24
 678         gateway 10.0.2.1   #this is the ip of the vnet2
 679         mtu 1450
 680 ----
 681
 682
 683 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
 684
 685 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
 686 will go to the configured 'myvnet2' gateway, then will be routed to gateway
 687 nodes ('node1' or 'node2') and from there it will leave those nodes over the
 688 default gateway configured on node1 or node2.
 689
 690 NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
 691 '10.0.2.0/24' network to node1, node2 on your external gateway, so that the
 692 public network can reply back.
 693
 694 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 695 and 10.0.2.0/24 in this example), will be announced dynamically.