]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[sysadmin_network_configuration]] |
0bcd1f7f DM |
2 | Network Configuration |
3 | --------------------- | |
5f09af76 DM |
4 | ifdef::wiki[] |
5 | :pve-toplevel: | |
6 | endif::wiki[] | |
7 | ||
0bcd1f7f DM |
8 | {pve} uses a bridged networking model. Each host can have up to 4094 |
9 | bridges. Bridges are like physical network switches implemented in | |
10 | software. All VMs can share a single bridge, as if | |
11 | virtual network cables from each guest were all plugged into the same | |
12 | switch. But you can also create multiple bridges to separate network | |
13 | domains. | |
14 | ||
15 | For connecting VMs to the outside world, bridges are attached to | |
16 | physical network cards. For further flexibility, you can configure | |
17 | VLANs (IEEE 802.1q) and network bonding, also known as "link | |
18 | aggregation". That way it is possible to build complex and flexible | |
19 | virtual networks. | |
20 | ||
8c1189b6 FG |
21 | Debian traditionally uses the `ifup` and `ifdown` commands to |
22 | configure the network. The file `/etc/network/interfaces` contains the | |
23 | whole network setup. Please refer to to manual page (`man interfaces`) | |
0bcd1f7f DM |
24 | for a complete format description. |
25 | ||
26 | NOTE: {pve} does not write changes directly to | |
8c1189b6 FG |
27 | `/etc/network/interfaces`. Instead, we write into a temporary file |
28 | called `/etc/network/interfaces.new`, and commit those changes when | |
0bcd1f7f DM |
29 | you reboot the node. |
30 | ||
31 | It is worth mentioning that you can directly edit the configuration | |
32 | file. All {pve} tools tries hard to keep such direct user | |
33 | modifications. Using the GUI is still preferable, because it | |
34 | protect you from errors. | |
35 | ||
5eba0743 | 36 | |
0bcd1f7f DM |
37 | Naming Conventions |
38 | ~~~~~~~~~~~~~~~~~~ | |
39 | ||
40 | We currently use the following naming conventions for device names: | |
41 | ||
7a0d4784 WL |
42 | * New Ethernet devices: en*, systemd network interface names. |
43 | ||
cc3cb912 | 44 | * Legacy Ethernet devices: eth[N], where 0 ≤ N (`eth0`, `eth1`, ...) |
7a0d4784 | 45 | They are available when Proxmox VE has been updated by an earlier version. |
0bcd1f7f DM |
46 | |
47 | * Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`) | |
48 | ||
49 | * Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...) | |
50 | ||
51 | * VLANs: Simply add the VLAN number to the device name, | |
7a0d4784 | 52 | separated by a period (`eno1.50`, `bond1.30`) |
0bcd1f7f DM |
53 | |
54 | This makes it easier to debug networks problems, because the device | |
55 | names implies the device type. | |
56 | ||
cc3cb912 | 57 | |
7a0d4784 WL |
58 | Systemd Network Interface Names |
59 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
60 | ||
8116dea5 DM |
61 | Systemd uses the two character prefix 'en' for Ethernet network |
62 | devices. The next characters depends on the device driver and the fact | |
cc3cb912 | 63 | which schema matches first. |
7a0d4784 WL |
64 | |
65 | * o<index>[n<phys_port_name>|d<dev_port>] — devices on board | |
66 | ||
67 | * s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id | |
68 | ||
69 | * [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id | |
70 | ||
71 | * x<MAC> — device by MAC address | |
72 | ||
cc3cb912 | 73 | The most common patterns are: |
7a0d4784 WL |
74 | |
75 | * eno1 — is the first on board NIC | |
76 | ||
77 | * enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1. | |
78 | ||
cc3cb912 DM |
79 | For more information see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/[Predictable Network Interface Names]. |
80 | ||
7a0d4784 | 81 | |
0bcd1f7f DM |
82 | Default Configuration using a Bridge |
83 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
84 | ||
85 | The installation program creates a single bridge named `vmbr0`, which | |
8116dea5 | 86 | is connected to the first Ethernet card `eno0`. The corresponding |
8c1189b6 | 87 | configuration in `/etc/network/interfaces` looks like this: |
0bcd1f7f DM |
88 | |
89 | ---- | |
90 | auto lo | |
91 | iface lo inet loopback | |
92 | ||
7a0d4784 | 93 | iface eno1 inet manual |
0bcd1f7f DM |
94 | |
95 | auto vmbr0 | |
96 | iface vmbr0 inet static | |
97 | address 192.168.10.2 | |
98 | netmask 255.255.255.0 | |
99 | gateway 192.168.10.1 | |
7a0d4784 | 100 | bridge_ports eno1 |
0bcd1f7f DM |
101 | bridge_stp off |
102 | bridge_fd 0 | |
103 | ---- | |
104 | ||
105 | Virtual machines behave as if they were directly connected to the | |
106 | physical network. The network, in turn, sees each virtual machine as | |
107 | having its own MAC, even though there is only one network cable | |
108 | connecting all of these VMs to the network. | |
109 | ||
110 | ||
111 | Routed Configuration | |
112 | ~~~~~~~~~~~~~~~~~~~~ | |
113 | ||
114 | Most hosting providers do not support the above setup. For security | |
115 | reasons, they disable networking as soon as they detect multiple MAC | |
116 | addresses on a single interface. | |
117 | ||
118 | TIP: Some providers allows you to register additional MACs on there | |
119 | management interface. This avoids the problem, but is clumsy to | |
120 | configure because you need to register a MAC for each of your VMs. | |
121 | ||
8c1189b6 | 122 | You can avoid the problem by ``routing'' all traffic via a single |
0bcd1f7f DM |
123 | interface. This makes sure that all network packets use the same MAC |
124 | address. | |
125 | ||
8c1189b6 | 126 | A common scenario is that you have a public IP (assume `192.168.10.2` |
0bcd1f7f | 127 | for this example), and an additional IP block for your VMs |
8c1189b6 | 128 | (`10.10.10.1/255.255.255.0`). We recommend the following setup for such |
0bcd1f7f DM |
129 | situations: |
130 | ||
131 | ---- | |
132 | auto lo | |
133 | iface lo inet loopback | |
134 | ||
7a0d4784 WL |
135 | auto eno1 |
136 | iface eno1 inet static | |
0bcd1f7f DM |
137 | address 192.168.10.2 |
138 | netmask 255.255.255.0 | |
139 | gateway 192.168.10.1 | |
1ed90852 | 140 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward |
7a0d4784 | 141 | post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp |
0bcd1f7f DM |
142 | |
143 | ||
144 | auto vmbr0 | |
145 | iface vmbr0 inet static | |
146 | address 10.10.10.1 | |
147 | netmask 255.255.255.0 | |
148 | bridge_ports none | |
149 | bridge_stp off | |
150 | bridge_fd 0 | |
151 | ---- | |
152 | ||
153 | ||
8c1189b6 FG |
154 | Masquerading (NAT) with `iptables` |
155 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
0bcd1f7f DM |
156 | |
157 | In some cases you may want to use private IPs behind your Proxmox | |
158 | host's true IP, and masquerade the traffic using NAT: | |
159 | ||
160 | ---- | |
161 | auto lo | |
162 | iface lo inet loopback | |
163 | ||
7a0d4784 | 164 | auto eno0 |
0bcd1f7f | 165 | #real IP adress |
7a0d4784 | 166 | iface eno1 inet static |
0bcd1f7f DM |
167 | address 192.168.10.2 |
168 | netmask 255.255.255.0 | |
169 | gateway 192.168.10.1 | |
170 | ||
171 | auto vmbr0 | |
172 | #private sub network | |
173 | iface vmbr0 inet static | |
174 | address 10.10.10.1 | |
175 | netmask 255.255.255.0 | |
176 | bridge_ports none | |
177 | bridge_stp off | |
178 | bridge_fd 0 | |
179 | ||
180 | post-up echo 1 > /proc/sys/net/ipv4/ip_forward | |
7a0d4784 WL |
181 | post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE |
182 | post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE | |
0bcd1f7f DM |
183 | ---- |
184 | ||
b4c06a93 WL |
185 | |
186 | Linux Bond | |
187 | ~~~~~~~~~~ | |
188 | ||
3eafe338 WL |
189 | Bonding (also called NIC teaming or Link Aggregation) is a technique |
190 | for binding multiple NIC's to a single network device. It is possible | |
191 | to achieve different goals, like make the network fault-tolerant, | |
192 | increase the performance or both together. | |
193 | ||
194 | High-speed hardware like Fibre Channel and the associated switching | |
195 | hardware can be quite expensive. By doing link aggregation, two NICs | |
196 | can appear as one logical interface, resulting in double speed. This | |
197 | is a native Linux kernel feature that is supported by most | |
198 | switches. If your nodes have multiple Ethernet ports, you can | |
199 | distribute your points of failure by running network cables to | |
200 | different switches and the bonded connection will failover to one | |
201 | cable or the other in case of network trouble. | |
202 | ||
203 | Aggregated links can improve live-migration delays and improve the | |
204 | speed of replication of data between Proxmox VE Cluster nodes. | |
b4c06a93 WL |
205 | |
206 | There are 7 modes for bonding: | |
207 | ||
208 | * *Round-robin (balance-rr):* Transmit network packets in sequential | |
209 | order from the first available network interface (NIC) slave through | |
210 | the last. This mode provides load balancing and fault tolerance. | |
211 | ||
212 | * *Active-backup (active-backup):* Only one NIC slave in the bond is | |
213 | active. A different slave becomes active if, and only if, the active | |
214 | slave fails. The single logical bonded interface's MAC address is | |
215 | externally visible on only one NIC (port) to avoid distortion in the | |
216 | network switch. This mode provides fault tolerance. | |
217 | ||
218 | * *XOR (balance-xor):* Transmit network packets based on [(source MAC | |
219 | address XOR'd with destination MAC address) modulo NIC slave | |
220 | count]. This selects the same NIC slave for each destination MAC | |
221 | address. This mode provides load balancing and fault tolerance. | |
222 | ||
223 | * *Broadcast (broadcast):* Transmit network packets on all slave | |
224 | network interfaces. This mode provides fault tolerance. | |
225 | ||
226 | * *IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):* Creates | |
227 | aggregation groups that share the same speed and duplex | |
228 | settings. Utilizes all slave network interfaces in the active | |
229 | aggregator group according to the 802.3ad specification. | |
230 | ||
231 | * *Adaptive transmit load balancing (balance-tlb):* Linux bonding | |
232 | driver mode that does not require any special network-switch | |
233 | support. The outgoing network packet traffic is distributed according | |
234 | to the current load (computed relative to the speed) on each network | |
235 | interface slave. Incoming traffic is received by one currently | |
236 | designated slave network interface. If this receiving slave fails, | |
237 | another slave takes over the MAC address of the failed receiving | |
238 | slave. | |
239 | ||
e60ce90c | 240 | * *Adaptive load balancing (balance-alb):* Includes balance-tlb plus receive |
b4c06a93 WL |
241 | load balancing (rlb) for IPV4 traffic, and does not require any |
242 | special network switch support. The receive load balancing is achieved | |
243 | by ARP negotiation. The bonding driver intercepts the ARP Replies sent | |
244 | by the local system on their way out and overwrites the source | |
245 | hardware address with the unique hardware address of one of the NIC | |
246 | slaves in the single logical bonded interface such that different | |
247 | network-peers use different MAC addresses for their network packet | |
248 | traffic. | |
249 | ||
250 | For the most setups the active-backup are the best choice or if your | |
251 | switch support LACP "IEEE 802.3ad" this mode should be preferred. | |
252 | ||
cd1de2c2 WL |
253 | The following bond configuration can be used as distributed/shared |
254 | storage network. The benefit would be that you get more speed and the | |
255 | network will be fault-tolerant. | |
256 | ||
b4c06a93 WL |
257 | .Example: Use bond with fixed IP address |
258 | ---- | |
259 | auto lo | |
260 | iface lo inet loopback | |
261 | ||
7a0d4784 | 262 | iface eno1 inet manual |
b4c06a93 | 263 | |
7a0d4784 | 264 | iface eno2 inet manual |
b4c06a93 WL |
265 | |
266 | auto bond0 | |
267 | iface bond0 inet static | |
7a0d4784 | 268 | slaves eno1 eno2 |
b4c06a93 WL |
269 | address 192.168.1.2 |
270 | netmask 255.255.255.0 | |
271 | bond_miimon 100 | |
272 | bond_mode 802.3ad | |
273 | bond_xmit_hash_policy layer2+3 | |
274 | ||
275 | auto vmbr0 | |
276 | iface vmbr0 inet static | |
277 | address 10.10.10.2 | |
278 | netmask 255.255.255.0 | |
279 | gateway 10.10.10.1 | |
7a0d4784 | 280 | bridge_ports eno1 |
b4c06a93 WL |
281 | bridge_stp off |
282 | bridge_fd 0 | |
283 | ||
284 | ---- | |
285 | ||
cd1de2c2 WL |
286 | |
287 | Another possibility it to use the bond directly as bridge port. | |
288 | This can be used to make the guest network fault-tolerant. | |
289 | ||
290 | .Example: Use a bond as bridge port | |
b4c06a93 WL |
291 | ---- |
292 | auto lo | |
293 | iface lo inet loopback | |
294 | ||
7a0d4784 | 295 | iface eno1 inet manual |
b4c06a93 | 296 | |
7a0d4784 | 297 | iface eno2 inet manual |
b4c06a93 WL |
298 | |
299 | auto bond0 | |
300 | iface bond0 inet maunal | |
7a0d4784 | 301 | slaves eno1 eno2 |
b4c06a93 WL |
302 | bond_miimon 100 |
303 | bond_mode 802.3ad | |
304 | bond_xmit_hash_policy layer2+3 | |
305 | ||
306 | auto vmbr0 | |
307 | iface vmbr0 inet static | |
308 | address 10.10.10.2 | |
309 | netmask 255.255.255.0 | |
310 | gateway 10.10.10.1 | |
311 | bridge_ports bond0 | |
312 | bridge_stp off | |
313 | bridge_fd 0 | |
314 | ||
315 | ---- | |
316 | ||
0bcd1f7f DM |
317 | //// |
318 | TODO: explain IPv6 support? | |
319 | TODO: explan OVS | |
320 | //// |