]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[sysadmin_network_configuration]] |
0bcd1f7f DM |
2 | Network Configuration |
3 | --------------------- | |
5f09af76 DM |
4 | ifdef::wiki[] |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
7 | ||
05213009 EK |
8 | Network configuration can be done either via the GUI, or by manually |
9 | editing the file `/etc/network/interfaces`, which contains the | |
10 | whole network configuration. The `interfaces(5)` manual page contains the | |
11 | complete format description. All {pve} tools try hard to keep direct | |
12 | user modifications, but using the GUI is still preferable, because it | |
13 | protects you from errors. | |
0bcd1f7f | 14 | |
05213009 EK |
15 | Once the network is configured, you can use the Debian traditional tools `ifup` |
16 | and `ifdown` commands to bring interfaces up and down. | |
0bcd1f7f DM |
17 | |
18 | NOTE: {pve} does not write changes directly to | |
8c1189b6 FG |
19 | `/etc/network/interfaces`. Instead, we write into a temporary file |
20 | called `/etc/network/interfaces.new`, and commit those changes when | |
0bcd1f7f DM |
21 | you reboot the node. |
22 | ||
0bcd1f7f DM |
23 | Naming Conventions |
24 | ~~~~~~~~~~~~~~~~~~ | |
25 | ||
26 | We currently use the following naming conventions for device names: | |
27 | ||
05213009 EK |
28 | * Ethernet devices: en*, systemd network interface names. This naming scheme is |
29 | used for new {pve} installations since version 5.0. | |
7a0d4784 | 30 | |
05213009 EK |
31 | * Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) This naming |
32 | scheme is used for {pve} hosts which were installed before the 5.0 | |
33 | release. When upgrading to 5.0, the names are kept as-is. | |
0bcd1f7f DM |
34 | |
35 | * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`) | |
36 | ||
37 | * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...) | |
38 | ||
39 | * VLANs: Simply add the VLAN number to the device name, | |
7a0d4784 | 40 | separated by a period (`eno1.50`, `bond1.30`) |
0bcd1f7f DM |
41 | |
42 | This makes it easier to debug networks problems, because the device | |
05213009 | 43 | name implies the device type. |
cc3cb912 | 44 | |
7a0d4784 WL |
45 | Systemd Network Interface Names |
46 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
47 | ||
8116dea5 DM |
48 | Systemd uses the two character prefix 'en' for Ethernet network |
49 | devices. The next characters depends on the device driver and the fact | |
cc3cb912 | 50 | which schema matches first. |
7a0d4784 WL |
51 | |
52 | * o<index>[n<phys_port_name>|d<dev_port>] — devices on board | |
53 | ||
54 | * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id | |
55 | ||
56 | * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id | |
57 | ||
58 | * x<MAC> — device by MAC address | |
59 | ||
cc3cb912 | 60 | The most common patterns are: |
7a0d4784 WL |
61 | |
62 | * eno1 — is the first on board NIC | |
63 | ||
64 | * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1. | |
65 | ||
cc3cb912 DM |
66 | For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names]. |
67 | ||
05213009 EK |
68 | Choosing a network configuration |
69 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
70 | ||
71 | Depending on your current network organization and your resources you can | |
72 | choose either a bridged, routed, or masquerading networking setup. | |
73 | ||
74 | {pve} server in a private LAN, using an external gateway to reach the internet | |
75 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
76 | ||
77 | The *Bridged* model makes the most sense in this case, and this is also | |
78 | the default mode on new {pve} installations. | |
79 | Each of your Guest system will have a virtual interface attached to the | |
80 | {pve} bridge. This is similar in effect to having the Guest network card | |
81 | directly connected to a new switch on your LAN, the {pve} host playing the role | |
82 | of the switch. | |
83 | ||
84 | {pve} server at hosting provider, with public IP ranges for Guests | |
85 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
86 | ||
87 | For this setup, you can use either a *Bridged* or *Routed* model, depending on | |
88 | what your provider allows. | |
89 | ||
90 | {pve} server at hosting provider, with a single public IP address | |
91 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
92 | ||
93 | In that case the only way to get outgoing network accesses for your guest | |
94 | systems is to use *Masquerading*. For incoming network access to your guests, | |
95 | you will need to configure *Port Forwarding*. | |
96 | ||
97 | For further flexibility, you can configure | |
98 | VLANs (IEEE 802.1q) and network bonding, also known as "link | |
99 | aggregation". That way it is possible to build complex and flexible | |
100 | virtual networks. | |
7a0d4784 | 101 | |
0bcd1f7f DM |
102 | Default Configuration using a Bridge |
103 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
104 | ||
05213009 EK |
105 | Bridges are like physical network switches implemented in software. |
106 | All VMs can share a single bridge, or you can create multiple bridges to | |
107 | separate network domains. Each host can have up to 4094 bridges. | |
108 | ||
0bcd1f7f | 109 | The installation program creates a single bridge named `vmbr0`, which |
04e8476d EK |
110 | is connected to the first Ethernet card. The corresponding |
111 | configuration in `/etc/network/interfaces` might look like this: | |
0bcd1f7f DM |
112 | |
113 | ---- | |
114 | auto lo | |
115 | iface lo inet loopback | |
116 | ||
7a0d4784 | 117 | iface eno1 inet manual |
0bcd1f7f DM |
118 | |
119 | auto vmbr0 | |
120 | iface vmbr0 inet static | |
121 | address 192.168.10.2 | |
122 | netmask 255.255.255.0 | |
123 | gateway 192.168.10.1 | |
7a0d4784 | 124 | bridge_ports eno1 |
0bcd1f7f DM |
125 | bridge_stp off |
126 | bridge_fd 0 | |
127 | ---- | |
128 | ||
129 | Virtual machines behave as if they were directly connected to the | |
130 | physical network. The network, in turn, sees each virtual machine as | |
131 | having its own MAC, even though there is only one network cable | |
132 | connecting all of these VMs to the network. | |
133 | ||
0bcd1f7f DM |
134 | Routed Configuration |
135 | ~~~~~~~~~~~~~~~~~~~~ | |
136 | ||
137 | Most hosting providers do not support the above setup. For security | |
138 | reasons, they disable networking as soon as they detect multiple MAC | |
139 | addresses on a single interface. | |
140 | ||
141 | TIP: Some providers allows you to register additional MACs on there | |
142 | management interface. This avoids the problem, but is clumsy to | |
143 | configure because you need to register a MAC for each of your VMs. | |
144 | ||
8c1189b6 | 145 | You can avoid the problem by ``routing'' all traffic via a single |
0bcd1f7f DM |
146 | interface. This makes sure that all network packets use the same MAC |
147 | address. | |
148 | ||
05213009 | 149 | A common scenario is that you have a public IP (assume `198.51.100.5` |
0bcd1f7f | 150 | for this example), and an additional IP block for your VMs |
05213009 | 151 | (`203.0.113.16/29`). We recommend the following setup for such |
0bcd1f7f DM |
152 | situations: |
153 | ||
154 | ---- | |
155 | auto lo | |
156 | iface lo inet loopback | |
157 | ||
7a0d4784 WL |
158 | auto eno1 |
159 | iface eno1 inet static | |
05213009 | 160 | address 198.51.100.5 |
0bcd1f7f | 161 | netmask 255.255.255.0 |
05213009 | 162 | gateway 198.51.100.1 |
1ed90852 | 163 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward |
7a0d4784 | 164 | post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp |
0bcd1f7f DM |
165 | |
166 | ||
167 | auto vmbr0 | |
168 | iface vmbr0 inet static | |
05213009 EK |
169 | address 203.0.113.17 |
170 | netmask 255.255.255.248 | |
0bcd1f7f DM |
171 | bridge_ports none |
172 | bridge_stp off | |
173 | bridge_fd 0 | |
174 | ---- | |
175 | ||
176 | ||
8c1189b6 FG |
177 | Masquerading (NAT) with `iptables` |
178 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
0bcd1f7f | 179 | |
05213009 EK |
180 | Masquerading allows guests having only a private IP address to access the |
181 | network by using the host IP address for outgoing traffic. Each outgoing | |
182 | packet is rewritten by `iptables` to appear as originating from the host, | |
183 | and responses are rewritten accordingly to be routed to the original sender. | |
0bcd1f7f DM |
184 | |
185 | ---- | |
186 | auto lo | |
187 | iface lo inet loopback | |
188 | ||
05213009 | 189 | auto eno1 |
470d4313 | 190 | #real IP address |
7a0d4784 | 191 | iface eno1 inet static |
05213009 | 192 | address 198.51.100.5 |
0bcd1f7f | 193 | netmask 255.255.255.0 |
05213009 | 194 | gateway 198.51.100.1 |
0bcd1f7f DM |
195 | |
196 | auto vmbr0 | |
197 | #private sub network | |
198 | iface vmbr0 inet static | |
199 | address 10.10.10.1 | |
200 | netmask 255.255.255.0 | |
201 | bridge_ports none | |
202 | bridge_stp off | |
203 | bridge_fd 0 | |
204 | ||
205 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward | |
7a0d4784 WL |
206 | post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE |
207 | post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE | |
0bcd1f7f DM |
208 | ---- |
209 | ||
b4c06a93 WL |
210 | |
211 | Linux Bond | |
212 | ~~~~~~~~~~ | |
213 | ||
3eafe338 WL |
214 | Bonding (also called NIC teaming or Link Aggregation) is a technique |
215 | for binding multiple NIC's to a single network device. It is possible | |
216 | to achieve different goals, like make the network fault-tolerant, | |
217 | increase the performance or both together. | |
218 | ||
219 | High-speed hardware like Fibre Channel and the associated switching | |
220 | hardware can be quite expensive. By doing link aggregation, two NICs | |
221 | can appear as one logical interface, resulting in double speed. This | |
222 | is a native Linux kernel feature that is supported by most | |
223 | switches. If your nodes have multiple Ethernet ports, you can | |
224 | distribute your points of failure by running network cables to | |
225 | different switches and the bonded connection will failover to one | |
226 | cable or the other in case of network trouble. | |
227 | ||
228 | Aggregated links can improve live-migration delays and improve the | |
229 | speed of replication of data between Proxmox VE Cluster nodes. | |
b4c06a93 WL |
230 | |
231 | There are 7 modes for bonding: | |
232 | ||
233 | * *Round-robin (balance-rr):* Transmit network packets in sequential | |
234 | order from the first available network interface (NIC) slave through | |
235 | the last. This mode provides load balancing and fault tolerance. | |
236 | ||
237 | * *Active-backup (active-backup):* Only one NIC slave in the bond is | |
238 | active. A different slave becomes active if, and only if, the active | |
239 | slave fails. The single logical bonded interface's MAC address is | |
240 | externally visible on only one NIC (port) to avoid distortion in the | |
241 | network switch. This mode provides fault tolerance. | |
242 | ||
243 | * *XOR (balance-xor):* Transmit network packets based on [(source MAC | |
244 | address XOR'd with destination MAC address) modulo NIC slave | |
245 | count]. This selects the same NIC slave for each destination MAC | |
246 | address. This mode provides load balancing and fault tolerance. | |
247 | ||
248 | * *Broadcast (broadcast):* Transmit network packets on all slave | |
249 | network interfaces. This mode provides fault tolerance. | |
250 | ||
251 | * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates | |
252 | aggregation groups that share the same speed and duplex | |
253 | settings. Utilizes all slave network interfaces in the active | |
254 | aggregator group according to the 802.3ad specification. | |
255 | ||
256 | * *Adaptive transmit load balancing (balance-tlb):* Linux bonding | |
257 | driver mode that does not require any special network-switch | |
258 | support. The outgoing network packet traffic is distributed according | |
259 | to the current load (computed relative to the speed) on each network | |
260 | interface slave. Incoming traffic is received by one currently | |
261 | designated slave network interface. If this receiving slave fails, | |
262 | another slave takes over the MAC address of the failed receiving | |
263 | slave. | |
264 | ||
e60ce90c | 265 | * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive |
b4c06a93 WL |
266 | load balancing (rlb) for IPV4 traffic, and does not require any |
267 | special network switch support. The receive load balancing is achieved | |
268 | by ARP negotiation. The bonding driver intercepts the ARP Replies sent | |
269 | by the local system on their way out and overwrites the source | |
270 | hardware address with the unique hardware address of one of the NIC | |
271 | slaves in the single logical bonded interface such that different | |
272 | network-peers use different MAC addresses for their network packet | |
273 | traffic. | |
274 | ||
649098a6 EK |
275 | If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using |
276 | the corresponding bonding mode (802.3ad). Otherwise you should generally use the | |
277 | active-backup mode. + | |
278 | // http://lists.linux-ha.org/pipermail/linux-ha/2013-January/046295.html | |
279 | If you intend to run your cluster network on the bonding interfaces, then you | |
280 | have to use active-passive mode on the bonding interfaces, other modes are | |
281 | unsupported. | |
b4c06a93 | 282 | |
cd1de2c2 WL |
283 | The following bond configuration can be used as distributed/shared |
284 | storage network. The benefit would be that you get more speed and the | |
285 | network will be fault-tolerant. | |
286 | ||
b4c06a93 WL |
287 | .Example: Use bond with fixed IP address |
288 | ---- | |
289 | auto lo | |
290 | iface lo inet loopback | |
291 | ||
7a0d4784 | 292 | iface eno1 inet manual |
b4c06a93 | 293 | |
7a0d4784 | 294 | iface eno2 inet manual |
b4c06a93 WL |
295 | |
296 | auto bond0 | |
297 | iface bond0 inet static | |
7a0d4784 | 298 | slaves eno1 eno2 |
b4c06a93 WL |
299 | address 192.168.1.2 |
300 | netmask 255.255.255.0 | |
301 | bond_miimon 100 | |
302 | bond_mode 802.3ad | |
303 | bond_xmit_hash_policy layer2+3 | |
304 | ||
305 | auto vmbr0 | |
306 | iface vmbr0 inet static | |
307 | address 10.10.10.2 | |
308 | netmask 255.255.255.0 | |
7ea42266 | 309 | gateway 10.10.10.1 |
7a0d4784 | 310 | bridge_ports eno1 |
b4c06a93 WL |
311 | bridge_stp off |
312 | bridge_fd 0 | |
313 | ||
314 | ---- | |
315 | ||
cd1de2c2 WL |
316 | |
317 | Another possibility it to use the bond directly as bridge port. | |
318 | This can be used to make the guest network fault-tolerant. | |
319 | ||
320 | .Example: Use a bond as bridge port | |
b4c06a93 WL |
321 | ---- |
322 | auto lo | |
323 | iface lo inet loopback | |
324 | ||
7a0d4784 | 325 | iface eno1 inet manual |
b4c06a93 | 326 | |
7a0d4784 | 327 | iface eno2 inet manual |
b4c06a93 WL |
328 | |
329 | auto bond0 | |
470d4313 | 330 | iface bond0 inet manual |
7a0d4784 | 331 | slaves eno1 eno2 |
b4c06a93 WL |
332 | bond_miimon 100 |
333 | bond_mode 802.3ad | |
334 | bond_xmit_hash_policy layer2+3 | |
335 | ||
336 | auto vmbr0 | |
337 | iface vmbr0 inet static | |
338 | address 10.10.10.2 | |
339 | netmask 255.255.255.0 | |
7ea42266 | 340 | gateway 10.10.10.1 |
b4c06a93 WL |
341 | bridge_ports bond0 |
342 | bridge_stp off | |
343 | bridge_fd 0 | |
344 | ||
345 | ---- | |
346 | ||
94fd8ea5 WL |
347 | VLAN 802.1Q |
348 | ~~~~~~~~~~~ | |
349 | ||
350 | A virtual LAN (VLAN) is any broadcast domain that is partitioned | |
351 | and isolated in the network at layer 2. | |
352 | So it is possible to have multiple networks (4096) in a physical network, | |
353 | each independent of the other ones. | |
354 | Each VLAN network is identified by a number often called `tag`. | |
355 | Network packages are then `tagged` to identify which virtual | |
356 | network they belong to. | |
357 | ||
358 | One or more VLANs can be used at any network device (Nic, Bond, Bridge). | |
359 | VLANs can be configured in several ways. Here, only the most common ones get | |
360 | described. We assume a network infrastructure based on Linux Kernel Networking | |
361 | (opposed to, e.g., Open vSwitch). | |
362 | Of course, there are scenarios that are not possible with this configuration, | |
363 | but it will work for most standard setups. | |
364 | ||
365 | Two of the most common and popular usage scenarios are: | |
366 | ||
367 | 1.) VLAN for the guest networks. | |
368 | Proxmox supports three different ways of using VLAN in guests: | |
369 | ||
370 | * *VLAN awareness on the Linux Bridge:* | |
371 | In this case, each guest's virtual network card is assigned to a VLAN tag, | |
372 | which is transparently supported by the Linux Bridge. | |
373 | Trunk mode is also possible, but that makes the configuration | |
374 | in the guest necessary. | |
375 | ||
376 | * *"traditional" VLAN on the Linux bridge:* | |
377 | In contrast to the VLAN awareness method, this method is not transparent | |
378 | and creates a VLAN device with associated bridge for each VLAN. | |
379 | That is, if e.g. in our default network, a guest VLAN 5 is used | |
380 | to create eno1.5 and vmbr0v5, which remains until rebooting. | |
381 | ||
382 | * *Guest configured:* The VLANs are assigned in the guest. | |
383 | In this case, the setup is in the guest and can not be influenced from the | |
384 | outside. | |
385 | The benefit is more then one VLAN on a single virtual NIC can be used. | |
386 | ||
387 | 2.) VLAN on the host, to allow the host communication whit an isolated network. | |
388 | As already mentioned, it is possible to apply the VLAN to all network devices. | |
389 | In general, you should configure the VLAN on the interface with the least | |
390 | abstraction layers between itself and the physical NIC. | |
391 | ||
392 | For example, in a default configuration where you want to place | |
393 | the host management address on a separate VLAN. | |
394 | ||
395 | NOTE: In the examples we use the VLAN at bridge level to ensure the correct | |
396 | function of VLAN 5 in the guest network, but in combination with VLAN anwareness | |
397 | bridge this it will not work for guest network VLAN 5. | |
398 | The downside of this setup is more CPU usage. | |
399 | ||
400 | .Example: Use VLAN 5 for the {pve} management IP | |
401 | ---- | |
402 | auto lo | |
403 | iface lo inet loopback | |
404 | ||
405 | iface eno1 inet manual | |
406 | ||
407 | iface eno1.5 inet manual | |
408 | ||
409 | auto vmbr0v5 | |
410 | iface vmbr0v5 inet static | |
411 | address 10.10.10.2 | |
412 | netmask 255.255.255.0 | |
413 | gateway 10.10.10.1 | |
414 | bridge_ports eno1.5 | |
415 | bridge_stp off | |
416 | bridge_fd 0 | |
417 | ||
418 | auto vmbr0 | |
419 | iface vmbr0 inet manual | |
420 | bridge_ports eno1 | |
421 | bridge_stp off | |
422 | bridge_fd 0 | |
423 | ||
424 | ---- | |
425 | ||
426 | The next example is the same setup but a bond is used to | |
427 | make this network fail-safe. | |
428 | ||
429 | .Example: Use VLAN 5 with bond0 for the {pve} management IP | |
430 | ---- | |
431 | auto lo | |
432 | iface lo inet loopback | |
433 | ||
434 | iface eno1 inet manual | |
435 | ||
436 | iface eno2 inet manual | |
437 | ||
438 | auto bond0 | |
439 | iface bond0 inet manual | |
440 | slaves eno1 eno2 | |
441 | bond_miimon 100 | |
442 | bond_mode 802.3ad | |
443 | bond_xmit_hash_policy layer2+3 | |
444 | ||
445 | iface bond0.5 inet manual | |
446 | ||
447 | auto vmbr0v5 | |
448 | iface vmbr0v5 inet static | |
449 | address 10.10.10.2 | |
450 | netmask 255.255.255.0 | |
451 | gateway 10.10.10.1 | |
452 | bridge_ports bond0.5 | |
453 | bridge_stp off | |
454 | bridge_fd 0 | |
455 | ||
456 | auto vmbr0 | |
457 | iface vmbr0 inet manual | |
458 | bridge_ports bond0 | |
459 | bridge_stp off | |
460 | bridge_fd 0 | |
461 | ||
462 | ---- | |
463 | ||
0bcd1f7f DM |
464 | //// |
465 | TODO: explain IPv6 support? | |
470d4313 | 466 | TODO: explain OVS |
0bcd1f7f | 467 | //// |