pvesdn.adoc

   1 [[chapter_pvesdn]]
   2 Software Defined Network
   3 ========================
   4 ifndef::manvolnum[]
   5 :pve-toplevel:
   6 endif::manvolnum[]
   7
   8 The **S**oftware **D**efined **N**etwork (SDN) feature allows one to create
   9 virtual networks (vnets) at datacenter level.
  10
  11 WARNING: SDN is currently an **experimental feature** in {pve}. This
  12 Documentation for it is also still under development, ask on our
  13 xref:getting_help[mailing lists or in the forum] for questions and feedback.
  14
  15
  16 [[pvesdn_installation]]
  17 Installation
  18 ------------
  19
  20 To enable the experimental SDN integration, you need to install the
  21 `libpve-network-perl` and `ifupdown2` package on every node:
  22
  23 ----
  24 apt update
  25 apt install libpve-network-perl ifupdown2
  26 ----
  27
  28 After that you need to add the following line:
  29
  30 ----
  31 source /etc/network/interfaces.d/*
  32 ----
  33 at the end of the `/etc/network/interfaces` configuration file, so that the SDN
  34 config gets included and activated.
  35
  36
  37 Basic Overview
  38 --------------
  39
  40 The {pve} SDN allows separation and fine grained control of Virtual Guests
  41 networks, using flexible software controlled configurations.
  42
  43 Separation consists of zones, a zone is it's own virtual separated network area.
  44 A 'VNet' is a type of a virtual network connected to a zone. Depending on which
  45 type or plugin the zone uses it can behave differently and offer different
  46 features, advantages or disadvantages.
  47 Normally a 'VNet' shows up as a common Linux bridge with either a VLAN or
  48 'VXLAN' tag, but some can also use layer 3 routing for control.
  49 The 'VNets' are deployed locally on each node, after configuration was committed
  50 from the cluster-wide datacenter SDN administration interface.
  51
  52
  53 Main configuration
  54 ~~~~~~~~~~~~~~~~~~
  55
  56 The configuration is done at datacenter (cluster-wide) level, it will be saved
  57 in configuration files located in the shared configuration file system:
  58 `/etc/pve/sdn`
  59
  60 On the web-interface  SDN feature have 3 main sections for the configuration
  61
  62 * SDN: a overview of the SDN state
  63
  64 * Zones: Create and manage the virtual separated network Zones
  65
  66 * VNets: Create virtual network bridges + subnets management.
  67
  68 And some options:
  69
  70 * Controller: For complex setups to control Layer 3 routing
  71
  72 * Sub-nets: Used to defined ip networks on VNets.
  73
  74 * IPAM: Allow to use external tools for IP address management (guest IPs)
  75
  76 * DNS: Allow to define a DNS server api for registering a virtual guests
  77   hostname and IP-addresses
  78
  79 [[pvesdn_config_main_sdn]]
  80
  81 SDN
  82 ~~~
  83
  84 This is the main status panel. Here you can see deployment status of zones on
  85 different nodes.
  86
  87 There is an 'Apply' button, to push and reload local configuration on all
  88 cluster nodes.
  89
  90
  91 [[pvesdn_local_deployment_monitoring]]
  92 Local Deployment Monitoring
  93 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  94
  95 After applying the configuration through the main SDN web-interface panel,
  96 the local network configuration is generated locally on each node in
  97 `/etc/network/interfaces.d/sdn`, and with ifupdown2 reloaded.
  98
  99 You can monitor the status of local zones and vnets through the main tree.
 100
 101
 102 [[pvesdn_config_zone]]
 103 Zones
 104 -----
 105
 106 A zone will define a virtually separated network.
 107
 108 It can use different technologies for separation:
 109
 110 * VLAN: Virtual LANs are the classic method to sub-divide a LAN
 111
 112 * QinQ: stacked VLAN (formally known as `IEEE 802.1ad`)
 113
 114 * VXLAN: (layer2 vxlan)
 115
 116 * Simple: Isolated Bridge, simple l3 routing bridge (NAT)
 117
 118 * bgp-evpn: vxlan using layer3 border gateway protocol routing
 119
 120 You can restrict a zone to specific nodes.
 121
 122 It's also possible to add permissions on a zone, to restrict user to use only a
 123 specific zone and only the VNets in that zone
 124
 125 Common options
 126 ~~~~~~~~~~~~~~
 127
 128 The following options are available for all zone types.
 129
 130 nodes:: Deploy and allow to use a VNets configured for this Zone only on these
 131 nodes.
 132
 133 ipam:: Optional, if you want to use an ipam tool to manage ips in  this zone
 134
 135 dns:: Optional, dns api server.
 136
 137 reversedns:: Optional, reverse dns api server.
 138
 139 dnszone:: Optional, dns domain name. Use to register hostname like
 140 `<hostname>.<domain>`. The dns zone need to be already existing in dns server.
 141
 142
 143 [[pvesdn_zone_plugin_simple]]
 144 Simple Zones
 145 ~~~~~~~~~~~~
 146
 147 This is the simplest plugin, it will create an isolated vnet bridge.
 148 This bridge is not linked to physical interfaces, VM traffic is only
 149 local to the node(s).
 150 It can be also used for NAT or routed setup.
 151
 152 [[pvesdn_zone_plugin_vlan]]
 153 VLAN Zones
 154 ~~~~~~~~~~
 155
 156 This plugin will reuse an existing local Linux or OVS bridge,
 157 and manage VLANs on it.
 158 The benefit of using SDN module, is that you can create different zones with
 159 specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
 160
 161 Specific `VLAN` configuration options:
 162
 163 bridge:: Reuse this local bridge or OVS switch, already
 164 configured on *each* local node.
 165
 166 [[pvesdn_zone_plugin_qinq]]
 167 QinQ Zones
 168 ~~~~~~~~~~
 169
 170 QinQ is stacked VLAN. The first VLAN tag defined for the zone
 171 (so called 'service-vlan'), and the second VLAN tag defined for the vnets
 172
 173 NOTE: Your physical network switches must support stacked VLANs!
 174
 175 Specific QinQ configuration options:
 176
 177 bridge:: A local VLAN-aware bridge already configured on each local node
 178
 179 service vlan:: The main VLAN tag of this zone
 180
 181 service vlan protocol:: allow to define a 802.1q (default) or 802.1ad service vlan type.
 182
 183 mtu:: Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
 184 For example, you reduce the MTU to `1496` if you physical interface MTU is
 185 `1500`.
 186
 187 [[pvesdn_zone_plugin_vxlan]]
 188 VXLAN Zones
 189 ~~~~~~~~~~~
 190
 191 The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
 192 network (named underlay). It encapsulate layer 2 Ethernet frames within layer
 193 4 UDP datagrams, using `4789` as the default destination port. You can, for
 194 example, create a private IPv4 VXLAN network on top of public internet network
 195 nodes.
 196 This is a layer2 tunnel only, no routing between different VNets is possible.
 197
 198 Each VNet will have use specific VXLAN id from the range (1 - 16777215).
 199
 200 Specific EVPN configuration options:
 201
 202 peers address list:: A list of IPs from all nodes through which you want to
 203 communicate. Can also be external nodes.
 204
 205 mtu:: Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
 206 lower than the outgoing physical interface.
 207
 208 [[pvesdn_zone_plugin_evpn]]
 209 EVPN Zones
 210 ~~~~~~~~~~
 211
 212 This is the most complex of all supported plugins.
 213
 214 BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
 215 have an anycast IP-address and or MAC-address. The bridge IP is the same on each
 216 node, with this a virtual guest can use that address as gateway.
 217
 218 Routing can work across VNets from different zones through a VRF (Virtual
 219 Routing and Forwarding) interface.
 220
 221 Specific EVPN configuration options:
 222
 223 VRF VXLAN tag:: This is a vxlan-id used for routing interconnect between vnets,
 224 it must be different than VXLAN-id of VNets
 225
 226 controller:: an EVPN-controller need to be defined first (see controller
 227 plugins section)
 228
 229 VNet MAC address:: A unique anycast MAC address for all VNets in this zone.
 230   Will be auto-generated if not defined.
 231
 232 Exit Nodes:: Optionnal. This is used if you want to define some proxmox nodes, as exit
 233   gateway from evpn network through real network. The configured nodes will
 234   announce a default route in the EVPN network.
 235
 236 Primary Exit Node:: Optionnal. If you use multiple exit-nodes, this force traffic
 237   to a primary exit-node instead loadbalancing on all nodes.
 238   This is required if you want to use Snat or if your upstream router don't support
 239   ecmp.
 240
 241 Exit Nodes local routing:: Optional. This is a special option if you need to
 242   reach a vm/ct service from an exit node. (By default, the exit nodes only
 243   allow forwarding traffic between real network and evpn network).
 244
 245 Advertise Subnets:: Optional. If you have silent vms/CT (for example, multiples
 246   ips by interfaces, and the anycast gateway don't see traffic from theses ips,
 247   the ips addresses won't be able to be reach inside the evpn network). This
 248   option will announce the full subnet in the evpn network in this case.
 249
 250 Disable Arp-Nd Suppression:: Optional. Don't suppression arp or nd packets.
 251   This is required if you use moving virtual ip in your guests vm.
 252   (Ip is moving but mac address change)
 253
 254 Route-target import:: Optional. Allow to import a list of external evpn route-targets.
 255   For Cross-DC or differents evpn networks interconnect.
 256
 257 MTU:: because VXLAN encapsulation use 50 bytes, the MTU needs to be 50 bytes
 258   lower than the maximal MTU of the outgoing physical interface.
 259
 260
 261 [[pvesdn_config_vnet]]
 262 VNets
 263 -----
 264
 265 A `VNet` is in its basic form just a Linux bridge that will be deployed locally
 266 on the node and used for Virtual Machine communication.
 267
 268 VNet properties are:
 269
 270 ID:: a 8 characters ID to name and identify a VNet
 271
 272 Alias:: Optional longer name, if the ID isn't enough
 273
 274 Zone:: The associated zone for this VNet
 275
 276 Tag:: The unique VLAN or VXLAN id
 277
 278 VLAN Aware:: Allow to add an extra VLAN tag in the virtual machine or
 279   container vNIC configurations or allow the guest OS to manage the VLAN's tag.
 280
 281 [[pvesdn_config_subnet]]
 282
 283 Sub-Nets
 284 ~~~~~~~~
 285
 286 A sub-network (subnet or sub-net) allows you to define a specific IP network
 287 (IPv4 or IPv6). For each VNET, you can define one or more subnets.
 288
 289 A subnet can be used to:
 290
 291 * restrict IP-addresses you can define on a specific VNET
 292 * assign routes/gateway on a VNET in layer 3 zones
 293 * enable SNAT on a VNET in layer 3 zones
 294 * auto assign IPs on virtual guests (VM or CT) through IPAM plugin
 295 * DNS registration through DNS plugins
 296
 297 If an IPAM server is associated to the subnet zone, the subnet prefix will be
 298 automatically registered in the IPAM.
 299
 300
 301 Subnet properties are:
 302
 303 ID:: a cidr network address. Ex: 10.0.0.0/8
 304
 305 Gateway:: ip address for the default gateway of the network.
 306           On layer3 zones (simple/evpn plugins), it'll be deployed on the vnet.
 307
 308 Snat:: Optional, Enable Snat for layer3 zones (simple/evpn plugins) for this subnet.
 309        The subnet source ip will be natted to server outgoing interface/ip.
 310        On evpn zone, it's done only on evpn gateway-nodes.
 311
 312 Dnszoneprefix:: Optional, add a prefix to domain registration, like <hostname>.prefix.<domain>
 313
 314
 315 [[pvesdn_config_controllers]]
 316 Controllers
 317 -----------
 318
 319 Some zone types need an external controller to manage the VNet control-plane.
 320 Currently this is only required for the `bgp-evpn` zone plugin.
 321
 322 [[pvesdn_controller_plugin_evpn]]
 323 EVPN Controller
 324 ~~~~~~~~~~~~~~~
 325
 326 For `BGP-EVPN`, we need a controller to manage the control plane.
 327 The currently supported software controller is the "frr" router.
 328 You may need to install it on each node where you want to deploy EVPN zones.
 329
 330 ----
 331 apt install frr frr-pythontools
 332 ----
 333
 334 Configuration options:
 335
 336 asn:: A unique BGP ASN number. It's highly recommended to use private ASN
 337 number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
 338 breaking, or get broken, by global routing by mistake.
 339
 340 peers:: An ip list of all nodes where you want to communicate for the EVPN (could be also
 341 external nodes or route reflectors servers)
 342
 343
 344 [[pvesdn_controller_plugin_BGP]]
 345 BGP Controller
 346 ~~~~~~~~~~~~~~~
 347
 348 The bgp controller is not used directly by a zone.
 349 You can used it to configure frr to manage bgp peers.
 350
 351 For BGP-evpn, it can be use to define a different ASN by node, so doing EBGP.
 352
 353 Configuration options:
 354
 355 node:: The node of this BGP controller
 356
 357 asn:: A unique BGP ASN number. It's highly recommended to use private ASN
 358   number from the range (64512 - 65534) or (4200000000 - 4294967294), as else
 359   you could end up breaking, or get broken, by global routing by mistake.
 360
 361 peers:: An IP list of peers you want to communicate with for the underlying
 362   BGP network.
 363
 364 ebgp:: If your peer's remote-AS is different, it's enabling EBGP.
 365
 366 loopback:: If you want to use a loopback or dummy interface as source for the
 367   evpn network. (for multipath)
 368
 369 ebgp-mutltihop:: if the peers are not directly connected or use loopback, you can increase the
 370   number of hops to reach them.
 371
 372 bgp-multipath-as-path-relax:: Allow to do ECMP if your peers have differents ASN.
 373
 374 [[pvesdn_config_ipam]]
 375 IPAMs
 376 -----
 377 IPAM (IP address management) tools, are used to manage/assign ips on your devices on the network.
 378 It can be used to find free ip address when you create a vm/ct for example (not yet implemented).
 379
 380 An IPAM is associated to 1 or multiple zones, to provide ip addresses for all subnets defined in this zone.
 381
 382
 383 [[pvesdn_ipam_plugin_pveipam]]
 384 {pve} IPAM plugin
 385 ~~~~~~~~~~~~~~~~~
 386
 387 This is the default internal IPAM for your proxmox cluster if you don't have
 388 external ipam software
 389
 390 [[pvesdn_ipam_plugin_phpipam]]
 391 phpIPAM plugin
 392 ~~~~~~~~~~~~~~
 393 https://phpipam.net/
 394
 395 You need to create an application in phpipam, and add an api token with admin
 396 permission
 397
 398 phpIPAM properties are:
 399
 400 url:: The REST-API endpoint: `http://phpipam.domain.com/api/<appname>/`
 401 token:: An API access token
 402 section:: An integer ID. Sections are group of subnets in phpIPAM. Default
 403  installations use `sectionid=1` for customers.
 404
 405 [[pvesdn_ipam_plugin_netbox]]
 406 Netbox IPAM plugin
 407 ~~~~~~~~~~~~~~~~~~
 408
 409 NetBox is an IP address management (IPAM) and data center infrastructure
 410 management (DCIM) tool, see the source code repository for details:
 411 https://github.com/netbox-community/netbox
 412
 413 You need to create an api token in netbox
 414 https://netbox.readthedocs.io/en/stable/api/authentication
 415
 416 NetBox properties are:
 417
 418 url:: The REST API endpoint: `http://yournetbox.domain.com/api`
 419 token:: An API access token
 420
 421 [[pvesdn_config_dns]]
 422 DNS
 423 ---
 424
 425 The DNS plugin in {pve} SDN is used to define a DNS API server for registration
 426 of your hostname and IP-address. A DNS configuration is associated with one or
 427 more zones, to provide DNS registration for all the sub-net IPs configured for
 428 a zone.
 429
 430 [[pvesdn_dns_plugin_powerdns]]
 431 PowerDNS plugin
 432 ~~~~~~~~~~~~~~~
 433 https://doc.powerdns.com/authoritative/http-api/index.html
 434
 435 You need to enable the webserver and the API in your PowerDNS config:
 436
 437 ----
 438 api=yes
 439 api-key=arandomgeneratedstring
 440 webserver=yes
 441 webserver-port=8081
 442 ----
 443
 444 Powerdns properties are:
 445
 446 url:: The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
 447 key:: An API access key
 448 ttl:: The default TTL for records
 449
 450
 451 Examples
 452 --------
 453
 454 [[pvesdn_setup_example_vlan]]
 455 VLAN Setup Example
 456 ~~~~~~~~~~~~~~~~~~
 457
 458 TIP: While we show plain configuration content here, almost everything should
 459 be configurable using the web-interface only.
 460
 461 Node1: /etc/network/interfaces
 462
 463 ----
 464 auto vmbr0
 465 iface vmbr0 inet manual
 466         bridge-ports eno1
 467         bridge-stp off
 468         bridge-fd 0
 469         bridge-vlan-aware yes
 470         bridge-vids 2-4094
 471
 472 #management ip on vlan100
 473 auto vmbr0.100
 474 iface vmbr0.100 inet static
 475         address 192.168.0.1/24
 476
 477 source /etc/network/interfaces.d/*
 478 ----
 479
 480 Node2: /etc/network/interfaces
 481
 482 ----
 483 auto vmbr0
 484 iface vmbr0 inet manual
 485         bridge-ports eno1
 486         bridge-stp off
 487         bridge-fd 0
 488         bridge-vlan-aware yes
 489         bridge-vids 2-4094
 490
 491 #management ip on vlan100
 492 auto vmbr0.100
 493 iface vmbr0.100 inet static
 494         address 192.168.0.2/24
 495
 496 source /etc/network/interfaces.d/*
 497 ----
 498
 499 Create a VLAN zone named `myvlanzone':
 500
 501 ----
 502 id: myvlanzone
 503 bridge: vmbr0
 504 ----
 505
 506 Create a VNet named `myvnet1' with `vlan-id` `10' and the previously created
 507 `myvlanzone' as it's zone.
 508
 509 ----
 510 id: myvnet1
 511 zone: myvlanzone
 512 tag: 10
 513 ----
 514
 515 Apply the configuration through the main SDN panel, to create VNets locally on
 516 each nodes.
 517
 518 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 519
 520 Use the following network configuration for this VM:
 521
 522 ----
 523 auto eth0
 524 iface eth0 inet static
 525         address 10.0.3.100/24
 526 ----
 527
 528 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 529 `myvnet1' as vm1.
 530
 531 Use the following network configuration for this VM:
 532
 533 ----
 534 auto eth0
 535 iface eth0 inet static
 536         address 10.0.3.101/24
 537 ----
 538
 539 Then, you should be able to ping between both VMs over that network.
 540
 541
 542 [[pvesdn_setup_example_qinq]]
 543 QinQ Setup Example
 544 ~~~~~~~~~~~~~~~~~~
 545
 546 TIP: While we show plain configuration content here, almost everything should
 547 be configurable using the web-interface only.
 548
 549 Node1: /etc/network/interfaces
 550
 551 ----
 552 auto vmbr0
 553 iface vmbr0 inet manual
 554         bridge-ports eno1
 555         bridge-stp off
 556         bridge-fd 0
 557         bridge-vlan-aware yes
 558         bridge-vids 2-4094
 559
 560 #management ip on vlan100
 561 auto vmbr0.100
 562 iface vmbr0.100 inet static
 563         address 192.168.0.1/24
 564
 565 source /etc/network/interfaces.d/*
 566 ----
 567
 568 Node2: /etc/network/interfaces
 569
 570 ----
 571 auto vmbr0
 572 iface vmbr0 inet manual
 573         bridge-ports eno1
 574         bridge-stp off
 575         bridge-fd 0
 576         bridge-vlan-aware yes
 577         bridge-vids 2-4094
 578
 579 #management ip on vlan100
 580 auto vmbr0.100
 581 iface vmbr0.100 inet static
 582         address 192.168.0.2/24
 583
 584 source /etc/network/interfaces.d/*
 585 ----
 586
 587 Create an QinQ zone named `qinqzone1' with service VLAN 20
 588
 589 ----
 590 id: qinqzone1
 591 bridge: vmbr0
 592 service vlan: 20
 593 ----
 594
 595 Create another QinQ zone named `qinqzone2' with service VLAN 30
 596
 597 ----
 598 id: qinqzone2
 599 bridge: vmbr0
 600 service vlan: 30
 601 ----
 602
 603 Create a VNet named `myvnet1' with customer vlan-id 100 on the previously
 604 created `qinqzone1' zone.
 605
 606 ----
 607 id: myvnet1
 608 zone: qinqzone1
 609 tag: 100
 610 ----
 611
 612 Create a `myvnet2' with customer VLAN-id 100 on the previously created
 613 `qinqzone2' zone.
 614
 615 ----
 616 id: myvnet2
 617 zone: qinqzone2
 618 tag: 100
 619 ----
 620
 621 Apply the configuration on the main SDN web-interface panel to create VNets
 622 locally on each nodes.
 623
 624 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 625
 626 Use the following network configuration for this VM:
 627
 628 ----
 629 auto eth0
 630 iface eth0 inet static
 631         address 10.0.3.100/24
 632 ----
 633
 634 Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
 635 `myvnet1' as vm1.
 636
 637 Use the following network configuration for this VM:
 638
 639 ----
 640 auto eth0
 641 iface eth0 inet static
 642         address 10.0.3.101/24
 643 ----
 644
 645 Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
 646 `myvnet2'.
 647
 648 Use the following network configuration for this VM:
 649
 650 ----
 651 auto eth0
 652 iface eth0 inet static
 653         address 10.0.3.102/24
 654 ----
 655
 656 Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
 657 `myvnet2' as vm3.
 658
 659 Use the following network configuration for this VM:
 660
 661 ----
 662 auto eth0
 663 iface eth0 inet static
 664         address 10.0.3.103/24
 665 ----
 666
 667 Then, you should be able to ping between the VMs 'vm1' and 'vm2', also
 668 between 'vm3' and 'vm4'. But, none of VMs 'vm1' or 'vm2' can ping the VMs 'vm3'
 669 or 'vm4', as they are on a different zone with different service-vlan.
 670
 671
 672 [[pvesdn_setup_example_vxlan]]
 673 VXLAN Setup Example
 674 ~~~~~~~~~~~~~~~~~~~
 675
 676 TIP: While we show plain configuration content here, almost everything should
 677 be configurable using the web-interface only.
 678
 679 node1: /etc/network/interfaces
 680
 681 ----
 682 auto vmbr0
 683 iface vmbr0 inet static
 684         address 192.168.0.1/24
 685         gateway 192.168.0.254
 686         bridge-ports eno1
 687         bridge-stp off
 688         bridge-fd 0
 689         mtu 1500
 690
 691 source /etc/network/interfaces.d/*
 692 ----
 693
 694 node2: /etc/network/interfaces
 695
 696 ----
 697 auto vmbr0
 698 iface vmbr0 inet static
 699         address 192.168.0.2/24
 700         gateway 192.168.0.254
 701         bridge-ports eno1
 702         bridge-stp off
 703         bridge-fd 0
 704         mtu 1500
 705
 706 source /etc/network/interfaces.d/*
 707 ----
 708
 709 node3: /etc/network/interfaces
 710
 711 ----
 712 auto vmbr0
 713 iface vmbr0 inet static
 714         address 192.168.0.3/24
 715         gateway 192.168.0.254
 716         bridge-ports eno1
 717         bridge-stp off
 718         bridge-fd 0
 719         mtu 1500
 720
 721 source /etc/network/interfaces.d/*
 722 ----
 723
 724 Create an VXLAN zone named `myvxlanzone', use the lower MTU to ensure the extra
 725 50 bytes of the VXLAN header can fit. Add all previously configured IPs from
 726 the nodes as peer address list.
 727
 728 ----
 729 id: myvxlanzone
 730 peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
 731 mtu: 1450
 732 ----
 733
 734 Create a VNet named `myvnet1' using the VXLAN zone `myvxlanzone' created
 735 previously.
 736
 737 ----
 738 id: myvnet1
 739 zone: myvxlanzone
 740 tag: 100000
 741 ----
 742
 743 Apply the configuration on the main SDN web-interface panel to create VNets
 744 locally on each nodes.
 745
 746 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 747
 748 Use the following network configuration for this VM, note the lower MTU here.
 749
 750 ----
 751 auto eth0
 752 iface eth0 inet static
 753         address 10.0.3.100/24
 754         mtu 1450
 755 ----
 756
 757 Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
 758 `myvnet1' as vm1.
 759
 760 Use the following network configuration for this VM:
 761
 762 ----
 763 auto eth0
 764 iface eth0 inet static
 765         address 10.0.3.101/24
 766         mtu 1450
 767 ----
 768
 769 Then, you should be able to ping between between 'vm1' and 'vm2'.
 770
 771
 772 [[pvesdn_setup_example_evpn]]
 773 EVPN Setup Example
 774 ~~~~~~~~~~~~~~~~~~
 775
 776 node1: /etc/network/interfaces
 777
 778 ----
 779 auto vmbr0
 780 iface vmbr0 inet static
 781         address 192.168.0.1/24
 782         gateway 192.168.0.254
 783         bridge-ports eno1
 784         bridge-stp off
 785         bridge-fd 0
 786         mtu 1500
 787
 788 source /etc/network/interfaces.d/*
 789 ----
 790
 791 node2: /etc/network/interfaces
 792
 793 ----
 794 auto vmbr0
 795 iface vmbr0 inet static
 796         address 192.168.0.2/24
 797         gateway 192.168.0.254
 798         bridge-ports eno1
 799         bridge-stp off
 800         bridge-fd 0
 801         mtu 1500
 802
 803 source /etc/network/interfaces.d/*
 804 ----
 805
 806 node3: /etc/network/interfaces
 807
 808 ----
 809 auto vmbr0
 810 iface vmbr0 inet static
 811         address 192.168.0.3/24
 812         gateway 192.168.0.254
 813         bridge-ports eno1
 814         bridge-stp off
 815         bridge-fd 0
 816         mtu 1500
 817
 818 source /etc/network/interfaces.d/*
 819 ----
 820
 821 Create a EVPN controller, using a private ASN number and above node addreesses
 822 as peers.
 823
 824 ----
 825 id: myevpnctl
 826 asn: 65000
 827 peers: 192.168.0.1,192.168.0.2,192.168.0.3
 828 ----
 829
 830 Create an EVPN zone named `myevpnzone' using the previously created
 831 EVPN-controller Define 'node1' and 'node2' as exit nodes.
 832
 833 ----
 834 id: myevpnzone
 835 vrf vxlan tag: 10000
 836 controller: myevpnctl
 837 mtu: 1450
 838 vnet mac address: 32:F4:05:FE:6C:0A
 839 exitnodes: node1,node2
 840 ----
 841
 842 Create the first VNet named `myvnet1' using the EVPN zone `myevpnzone'.
 843 ----
 844 id: myvnet1
 845 zone: myevpnzone
 846 tag: 11000
 847 ----
 848
 849 Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on vnet1
 850
 851 ----
 852 subnet: 10.0.1.0/24
 853 gateway: 10.0.1.1
 854 ----
 855
 856 Create the second VNet named `myvnet2' using the same EVPN zone `myevpnzone', a
 857 different IPv4 CIDR network.
 858
 859 ----
 860 id: myvnet2
 861 zone: myevpnzone
 862 tag: 12000
 863 ----
 864
 865 Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2
 866
 867 ----
 868 subnet: 10.0.2.0/24
 869 gateway: 10.0.2.1
 870 ----
 871
 872
 873 Apply the configuration on the main SDN web-interface panel to create VNets
 874 locally on each nodes and generate the FRR config.
 875
 876 Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on `myvnet1'.
 877
 878 Use the following network configuration for this VM:
 879
 880 ----
 881 auto eth0
 882 iface eth0 inet static
 883         address 10.0.1.100/24
 884         gateway 10.0.1.1   #this is the ip of the vnet1
 885         mtu 1450
 886 ----
 887
 888 Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
 889 `myvnet2'.
 890
 891 Use the following network configuration for this VM:
 892
 893 ----
 894 auto eth0
 895 iface eth0 inet static
 896         address 10.0.2.100/24
 897         gateway 10.0.2.1   #this is the ip of the vnet2
 898         mtu 1450
 899 ----
 900
 901
 902 Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
 903
 904 If you ping an external IP from 'vm2' on the non-gateway 'node3', the packet
 905 will go to the configured 'myvnet2' gateway, then will be routed to the exit
 906 nodes ('node1' or 'node2') and from there it will leave those nodes over the
 907 default gateway configured on node1 or node2.
 908
 909 NOTE: Of course you need to add reverse routes for the '10.0.1.0/24' and
 910 '10.0.2.0/24' network to node1, node2 on your external gateway, so that the
 911 public network can reply back.
 912
 913 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 914 and 10.0.2.0/24 in this example), will be announced dynamically.
 915
 916
 917 Notes
 918 -----
 919
 920 VXLAN IPSEC Encryption
 921 ~~~~~~~~~~~~~~~~~~~~~~
 922 If you need to add encryption on top of VXLAN, it's possible to do so with
 923 IPSEC through `strongswan`. You'll need to reduce the 'MTU' by 60 bytes (IPv4)
 924 or 80 bytes (IPv6) to handle encryption.
 925
 926 So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
 927 + 50 (VXLAN) == 1500).
 928
 929 .Install strongswan
 930 ----
 931 apt install strongswan
 932 ----
 933
 934 Add configuration in `/etc/ipsec.conf'. We only need to encrypt traffic from
 935 the VXLAN UDP port '4789'.
 936
 937 ----
 938 conn %default
 939     ike=aes256-sha1-modp1024!  # the fastest, but reasonably secure cipher on modern HW
 940     esp=aes256-sha1!
 941     leftfirewall=yes           # this is necessary when using Proxmox VE firewall rules
 942
 943 conn output
 944     rightsubnet=%dynamic[udp/4789]
 945     right=%any
 946     type=transport
 947     authby=psk
 948     auto=route
 949
 950 conn input
 951     leftsubnet=%dynamic[udp/4789]
 952     type=transport
 953     authby=psk
 954     auto=route
 955 ----
 956
 957 Then generate a preshared key with
 958
 959 ----
 960 openssl rand -base64 128
 961 ----
 962
 963 and copy the key in `/etc/ipsec.secrets' so that the file content looks like:
 964
 965 ----
 966 : PSK <generatedbase64key>
 967 ----
 968
 969 You need to copy the PSK and the config on other nodes.