]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[sysadmin_network_configuration]] |
0bcd1f7f DM |
2 | Network Configuration |
3 | --------------------- | |
5f09af76 DM |
4 | ifdef::wiki[] |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
7 | ||
0bcd1f7f DM |
8 | {pve} uses a bridged networking model. Each host can have up to 4094 |
9 | bridges. Bridges are like physical network switches implemented in | |
10 | software. All VMs can share a single bridge, as if | |
11 | virtual network cables from each guest were all plugged into the same | |
12 | switch. But you can also create multiple bridges to separate network | |
13 | domains. | |
14 | ||
15 | For connecting VMs to the outside world, bridges are attached to | |
16 | physical network cards. For further flexibility, you can configure | |
17 | VLANs (IEEE 802.1q) and network bonding, also known as "link | |
18 | aggregation". That way it is possible to build complex and flexible | |
19 | virtual networks. | |
20 | ||
8c1189b6 FG |
21 | Debian traditionally uses the `ifup` and `ifdown` commands to |
22 | configure the network. The file `/etc/network/interfaces` contains the | |
23 | whole network setup. Please refer to to manual page (`man interfaces`) | |
0bcd1f7f DM |
24 | for a complete format description. |
25 | ||
26 | NOTE: {pve} does not write changes directly to | |
8c1189b6 FG |
27 | `/etc/network/interfaces`. Instead, we write into a temporary file |
28 | called `/etc/network/interfaces.new`, and commit those changes when | |
0bcd1f7f DM |
29 | you reboot the node. |
30 | ||
31 | It is worth mentioning that you can directly edit the configuration | |
32 | file. All {pve} tools tries hard to keep such direct user | |
33 | modifications. Using the GUI is still preferable, because it | |
34 | protect you from errors. | |
35 | ||
5eba0743 | 36 | |
0bcd1f7f DM |
37 | Naming Conventions |
38 | ~~~~~~~~~~~~~~~~~~ | |
39 | ||
40 | We currently use the following naming conventions for device names: | |
41 | ||
7a0d4784 WL |
42 | * New Ethernet devices: en*, systemd network interface names. |
43 | ||
44 | * Lagacy Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) | |
45 | They are available when Proxmox VE has been updated by an earlier version. | |
0bcd1f7f DM |
46 | |
47 | * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`) | |
48 | ||
49 | * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...) | |
50 | ||
51 | * VLANs: Simply add the VLAN number to the device name, | |
7a0d4784 | 52 | separated by a period (`eno1.50`, `bond1.30`) |
0bcd1f7f DM |
53 | |
54 | This makes it easier to debug networks problems, because the device | |
55 | names implies the device type. | |
56 | ||
7a0d4784 WL |
57 | Systemd Network Interface Names |
58 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
59 | ||
60 | Two character prefixes based on the type of interface: | |
61 | ||
62 | * en — Enoernet | |
63 | ||
64 | * sl — serial line IP (slip) | |
65 | ||
66 | * wl — wlan | |
67 | ||
68 | * ww — wwan | |
69 | ||
70 | The next characters depence on the device driver and the fact which schema matches first. | |
71 | ||
72 | * o<index>[n<phys_port_name>|d<dev_port>] — devices on board | |
73 | ||
74 | * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id | |
75 | ||
76 | * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id | |
77 | ||
78 | * x<MAC> — device by MAC address | |
79 | ||
80 | The most common patterns are | |
81 | ||
82 | * eno1 — is the first on board NIC | |
83 | ||
84 | * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1. | |
85 | ||
86 | For more information see link:https://github.com/systemd/systemd/blob/master/src/udev/udev-builtin-net_id.c#L20[Systemd Network Interface Names] | |
87 | ||
0bcd1f7f DM |
88 | Default Configuration using a Bridge |
89 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
90 | ||
91 | The installation program creates a single bridge named `vmbr0`, which | |
7a0d4784 | 92 | is connected to the first ethernet card `eno0`. The corresponding |
8c1189b6 | 93 | configuration in `/etc/network/interfaces` looks like this: |
0bcd1f7f DM |
94 | |
95 | ---- | |
96 | auto lo | |
97 | iface lo inet loopback | |
98 | ||
7a0d4784 | 99 | iface eno1 inet manual |
0bcd1f7f DM |
100 | |
101 | auto vmbr0 | |
102 | iface vmbr0 inet static | |
103 | address 192.168.10.2 | |
104 | netmask 255.255.255.0 | |
105 | gateway 192.168.10.1 | |
7a0d4784 | 106 | bridge_ports eno1 |
0bcd1f7f DM |
107 | bridge_stp off |
108 | bridge_fd 0 | |
109 | ---- | |
110 | ||
111 | Virtual machines behave as if they were directly connected to the | |
112 | physical network. The network, in turn, sees each virtual machine as | |
113 | having its own MAC, even though there is only one network cable | |
114 | connecting all of these VMs to the network. | |
115 | ||
116 | ||
117 | Routed Configuration | |
118 | ~~~~~~~~~~~~~~~~~~~~ | |
119 | ||
120 | Most hosting providers do not support the above setup. For security | |
121 | reasons, they disable networking as soon as they detect multiple MAC | |
122 | addresses on a single interface. | |
123 | ||
124 | TIP: Some providers allows you to register additional MACs on there | |
125 | management interface. This avoids the problem, but is clumsy to | |
126 | configure because you need to register a MAC for each of your VMs. | |
127 | ||
8c1189b6 | 128 | You can avoid the problem by ``routing'' all traffic via a single |
0bcd1f7f DM |
129 | interface. This makes sure that all network packets use the same MAC |
130 | address. | |
131 | ||
8c1189b6 | 132 | A common scenario is that you have a public IP (assume `192.168.10.2` |
0bcd1f7f | 133 | for this example), and an additional IP block for your VMs |
8c1189b6 | 134 | (`10.10.10.1/255.255.255.0`). We recommend the following setup for such |
0bcd1f7f DM |
135 | situations: |
136 | ||
137 | ---- | |
138 | auto lo | |
139 | iface lo inet loopback | |
140 | ||
7a0d4784 WL |
141 | auto eno1 |
142 | iface eno1 inet static | |
0bcd1f7f DM |
143 | address 192.168.10.2 |
144 | netmask 255.255.255.0 | |
145 | gateway 192.168.10.1 | |
7a0d4784 | 146 | post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp |
0bcd1f7f DM |
147 | |
148 | ||
149 | auto vmbr0 | |
150 | iface vmbr0 inet static | |
151 | address 10.10.10.1 | |
152 | netmask 255.255.255.0 | |
153 | bridge_ports none | |
154 | bridge_stp off | |
155 | bridge_fd 0 | |
156 | ---- | |
157 | ||
158 | ||
8c1189b6 FG |
159 | Masquerading (NAT) with `iptables` |
160 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
0bcd1f7f DM |
161 | |
162 | In some cases you may want to use private IPs behind your Proxmox | |
163 | host's true IP, and masquerade the traffic using NAT: | |
164 | ||
165 | ---- | |
166 | auto lo | |
167 | iface lo inet loopback | |
168 | ||
7a0d4784 | 169 | auto eno0 |
0bcd1f7f | 170 | #real IP adress |
7a0d4784 | 171 | iface eno1 inet static |
0bcd1f7f DM |
172 | address 192.168.10.2 |
173 | netmask 255.255.255.0 | |
174 | gateway 192.168.10.1 | |
175 | ||
176 | auto vmbr0 | |
177 | #private sub network | |
178 | iface vmbr0 inet static | |
179 | address 10.10.10.1 | |
180 | netmask 255.255.255.0 | |
181 | bridge_ports none | |
182 | bridge_stp off | |
183 | bridge_fd 0 | |
184 | ||
185 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward | |
7a0d4784 WL |
186 | post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE |
187 | post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE | |
0bcd1f7f DM |
188 | ---- |
189 | ||
b4c06a93 WL |
190 | |
191 | Linux Bond | |
192 | ~~~~~~~~~~ | |
193 | ||
3eafe338 WL |
194 | Bonding (also called NIC teaming or Link Aggregation) is a technique |
195 | for binding multiple NIC's to a single network device. It is possible | |
196 | to achieve different goals, like make the network fault-tolerant, | |
197 | increase the performance or both together. | |
198 | ||
199 | High-speed hardware like Fibre Channel and the associated switching | |
200 | hardware can be quite expensive. By doing link aggregation, two NICs | |
201 | can appear as one logical interface, resulting in double speed. This | |
202 | is a native Linux kernel feature that is supported by most | |
203 | switches. If your nodes have multiple Ethernet ports, you can | |
204 | distribute your points of failure by running network cables to | |
205 | different switches and the bonded connection will failover to one | |
206 | cable or the other in case of network trouble. | |
207 | ||
208 | Aggregated links can improve live-migration delays and improve the | |
209 | speed of replication of data between Proxmox VE Cluster nodes. | |
b4c06a93 WL |
210 | |
211 | There are 7 modes for bonding: | |
212 | ||
213 | * *Round-robin (balance-rr):* Transmit network packets in sequential | |
214 | order from the first available network interface (NIC) slave through | |
215 | the last. This mode provides load balancing and fault tolerance. | |
216 | ||
217 | * *Active-backup (active-backup):* Only one NIC slave in the bond is | |
218 | active. A different slave becomes active if, and only if, the active | |
219 | slave fails. The single logical bonded interface's MAC address is | |
220 | externally visible on only one NIC (port) to avoid distortion in the | |
221 | network switch. This mode provides fault tolerance. | |
222 | ||
223 | * *XOR (balance-xor):* Transmit network packets based on [(source MAC | |
224 | address XOR'd with destination MAC address) modulo NIC slave | |
225 | count]. This selects the same NIC slave for each destination MAC | |
226 | address. This mode provides load balancing and fault tolerance. | |
227 | ||
228 | * *Broadcast (broadcast):* Transmit network packets on all slave | |
229 | network interfaces. This mode provides fault tolerance. | |
230 | ||
231 | * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates | |
232 | aggregation groups that share the same speed and duplex | |
233 | settings. Utilizes all slave network interfaces in the active | |
234 | aggregator group according to the 802.3ad specification. | |
235 | ||
236 | * *Adaptive transmit load balancing (balance-tlb):* Linux bonding | |
237 | driver mode that does not require any special network-switch | |
238 | support. The outgoing network packet traffic is distributed according | |
239 | to the current load (computed relative to the speed) on each network | |
240 | interface slave. Incoming traffic is received by one currently | |
241 | designated slave network interface. If this receiving slave fails, | |
242 | another slave takes over the MAC address of the failed receiving | |
243 | slave. | |
244 | ||
e60ce90c | 245 | * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive |
b4c06a93 WL |
246 | load balancing (rlb) for IPV4 traffic, and does not require any |
247 | special network switch support. The receive load balancing is achieved | |
248 | by ARP negotiation. The bonding driver intercepts the ARP Replies sent | |
249 | by the local system on their way out and overwrites the source | |
250 | hardware address with the unique hardware address of one of the NIC | |
251 | slaves in the single logical bonded interface such that different | |
252 | network-peers use different MAC addresses for their network packet | |
253 | traffic. | |
254 | ||
255 | For the most setups the active-backup are the best choice or if your | |
256 | switch support LACP "IEEE 802.3ad" this mode should be preferred. | |
257 | ||
cd1de2c2 WL |
258 | The following bond configuration can be used as distributed/shared |
259 | storage network. The benefit would be that you get more speed and the | |
260 | network will be fault-tolerant. | |
261 | ||
b4c06a93 WL |
262 | .Example: Use bond with fixed IP address |
263 | ---- | |
264 | auto lo | |
265 | iface lo inet loopback | |
266 | ||
7a0d4784 | 267 | iface eno1 inet manual |
b4c06a93 | 268 | |
7a0d4784 | 269 | iface eno2 inet manual |
b4c06a93 WL |
270 | |
271 | auto bond0 | |
272 | iface bond0 inet static | |
7a0d4784 | 273 | slaves eno1 eno2 |
b4c06a93 WL |
274 | address 192.168.1.2 |
275 | netmask 255.255.255.0 | |
276 | bond_miimon 100 | |
277 | bond_mode 802.3ad | |
278 | bond_xmit_hash_policy layer2+3 | |
279 | ||
280 | auto vmbr0 | |
281 | iface vmbr0 inet static | |
282 | address 10.10.10.2 | |
283 | netmask 255.255.255.0 | |
284 | gateway 10.10.10.1 | |
7a0d4784 | 285 | bridge_ports eno1 |
b4c06a93 WL |
286 | bridge_stp off |
287 | bridge_fd 0 | |
288 | ||
289 | ---- | |
290 | ||
cd1de2c2 WL |
291 | |
292 | Another possibility it to use the bond directly as bridge port. | |
293 | This can be used to make the guest network fault-tolerant. | |
294 | ||
295 | .Example: Use a bond as bridge port | |
b4c06a93 WL |
296 | ---- |
297 | auto lo | |
298 | iface lo inet loopback | |
299 | ||
7a0d4784 | 300 | iface eno1 inet manual |
b4c06a93 | 301 | |
7a0d4784 | 302 | iface eno2 inet manual |
b4c06a93 WL |
303 | |
304 | auto bond0 | |
305 | iface bond0 inet maunal | |
7a0d4784 | 306 | slaves eno1 eno2 |
b4c06a93 WL |
307 | bond_miimon 100 |
308 | bond_mode 802.3ad | |
309 | bond_xmit_hash_policy layer2+3 | |
310 | ||
311 | auto vmbr0 | |
312 | iface vmbr0 inet static | |
313 | address 10.10.10.2 | |
314 | netmask 255.255.255.0 | |
315 | gateway 10.10.10.1 | |
316 | bridge_ports bond0 | |
317 | bridge_stp off | |
318 | bridge_fd 0 | |
319 | ||
320 | ---- | |
321 | ||
0bcd1f7f DM |
322 | //// |
323 | TODO: explain IPv6 support? | |
324 | TODO: explan OVS | |
325 | //// |