]>
Commit | Line | Data |
---|---|---|
542cc9bb TG |
1 | Design Decisions In Open vSwitch |
2 | ================================ | |
d31f1109 JP |
3 | |
4 | This document describes design decisions that went into implementing | |
5 | Open vSwitch. While we believe these to be reasonable decisions, it is | |
6 | impossible to predict how Open vSwitch will be used in all environments. | |
7 | Understanding assumptions made by Open vSwitch is critical to a | |
8 | successful deployment. The end of this document contains contact | |
9 | information that can be used to let us know how we can make Open vSwitch | |
10 | more generally useful. | |
11 | ||
80d5aefd BP |
12 | Asynchronous Messages |
13 | ===================== | |
14 | ||
15 | Over time, Open vSwitch has added many knobs that control whether a | |
16 | given controller receives OpenFlow asynchronous messages. This | |
17 | section describes how all of these features interact. | |
18 | ||
19 | First, a service controller never receives any asynchronous messages | |
4550b647 MM |
20 | unless it changes its miss_send_len from the service controller |
21 | default of zero in one of the following ways: | |
22 | ||
542cc9bb | 23 | - Sending an OFPT_SET_CONFIG message with nonzero miss_send_len. |
4550b647 | 24 | |
542cc9bb TG |
25 | - Sending any NXT_SET_ASYNC_CONFIG message: as a side effect, this |
26 | message changes the miss_send_len to | |
27 | OFP_DEFAULT_MISS_SEND_LEN (128) for service controllers. | |
80d5aefd BP |
28 | |
29 | Second, OFPT_FLOW_REMOVED and NXT_FLOW_REMOVED messages are generated | |
30 | only if the flow that was removed had the OFPFF_SEND_FLOW_REM flag | |
31 | set. | |
32 | ||
a7349929 BP |
33 | Third, OFPT_PACKET_IN and NXT_PACKET_IN messages are sent only to |
34 | OpenFlow controller connections that have the correct connection ID | |
35 | (see "struct nx_controller_id" and "struct nx_action_controller"): | |
36 | ||
542cc9bb TG |
37 | - For packet-in messages generated by a NXAST_CONTROLLER action, |
38 | the controller ID specified in the action. | |
a7349929 | 39 | |
542cc9bb TG |
40 | - For other packet-in messages, controller ID zero. (This is the |
41 | default ID when an OpenFlow controller does not configure one.) | |
a7349929 | 42 | |
80d5aefd BP |
43 | Finally, Open vSwitch consults a per-connection table indexed by the |
44 | message type, reason code, and current role. The following table | |
45 | shows how this table is initialized by default when an OpenFlow | |
46 | connection is made. An entry labeled "yes" means that the message is | |
47 | sent, an entry labeled "---" means that the message is suppressed. | |
48 | ||
542cc9bb | 49 | ``` |
80d5aefd BP |
50 | master/ |
51 | message and reason code other slave | |
52 | ---------------------------------------- ------- ----- | |
53 | OFPT_PACKET_IN / NXT_PACKET_IN | |
54 | OFPR_NO_MATCH yes --- | |
55 | OFPR_ACTION yes --- | |
56 | OFPR_INVALID_TTL --- --- | |
3a11fd5b | 57 | OFPR_GROUP (OF1.4+) yes --- |
80d5aefd BP |
58 | |
59 | OFPT_FLOW_REMOVED / NXT_FLOW_REMOVED | |
60 | OFPRR_IDLE_TIMEOUT yes --- | |
61 | OFPRR_HARD_TIMEOUT yes --- | |
62 | OFPRR_DELETE yes --- | |
63 | ||
64 | OFPT_PORT_STATUS | |
65 | OFPPR_ADD yes yes | |
66 | OFPPR_DELETE yes yes | |
67 | OFPPR_MODIFY yes yes | |
542cc9bb | 68 | ``` |
80d5aefd BP |
69 | |
70 | The NXT_SET_ASYNC_CONFIG message directly sets all of the values in | |
71 | this table for the current connection. The | |
72 | OFPC_INVALID_TTL_TO_CONTROLLER bit in the OFPT_SET_CONFIG message | |
73 | controls the setting for OFPR_INVALID_TTL for the "master" role. | |
74 | ||
75 | ||
76 | OFPAT_ENQUEUE | |
77 | ============= | |
82172632 EJ |
78 | |
79 | The OpenFlow 1.0 specification requires the output port of the OFPAT_ENQUEUE | |
80 | action to "refer to a valid physical port (i.e. < OFPP_MAX) or OFPP_IN_PORT". | |
81 | Although OFPP_LOCAL is not less than OFPP_MAX, it is an 'internal' port which | |
82 | can have QoS applied to it in Linux. Since we allow the OFPAT_ENQUEUE to apply | |
83 | to 'internal' ports whose port numbers are less than OFPP_MAX, we interpret | |
84 | OFPP_LOCAL as a physical port and support OFPAT_ENQUEUE on it as well. | |
85 | ||
d31f1109 | 86 | |
12442ec5 BP |
87 | OFPT_FLOW_MOD |
88 | ============= | |
89 | ||
3432cb4e BP |
90 | The OpenFlow specification for the behavior of OFPT_FLOW_MOD is |
91 | confusing. The following tables summarize the Open vSwitch | |
12442ec5 BP |
92 | implementation of its behavior in the following categories: |
93 | ||
542cc9bb TG |
94 | - "match on priority": Whether the flow_mod acts only on flows |
95 | whose priority matches that included in the flow_mod message. | |
12442ec5 | 96 | |
542cc9bb TG |
97 | - "match on out_port": Whether the flow_mod acts only on flows |
98 | that output to the out_port included in the flow_mod message (if | |
99 | out_port is not OFPP_NONE). OpenFlow 1.1 and later have a | |
100 | similar feature (not listed separately here) for out_group. | |
3432cb4e | 101 | |
542cc9bb TG |
102 | - "match on flow_cookie": Whether the flow_mod acts only on flows |
103 | whose flow_cookie matches an optional controller-specified value | |
104 | and mask. | |
12442ec5 | 105 | |
542cc9bb TG |
106 | - "updates flow_cookie": Whether the flow_mod changes the |
107 | flow_cookie of the flow or flows that it matches to the | |
108 | flow_cookie included in the flow_mod message. | |
12442ec5 | 109 | |
542cc9bb TG |
110 | - "updates OFPFF_ flags": Whether the flow_mod changes the |
111 | OFPFF_SEND_FLOW_REM flag of the flow or flows that it matches to | |
112 | the setting included in the flags of the flow_mod message. | |
12442ec5 | 113 | |
542cc9bb TG |
114 | - "honors OFPFF_CHECK_OVERLAP": Whether the OFPFF_CHECK_OVERLAP |
115 | flag in the flow_mod is significant. | |
12442ec5 | 116 | |
542cc9bb TG |
117 | - "updates idle_timeout" and "updates hard_timeout": Whether the |
118 | idle_timeout and hard_timeout in the flow_mod, respectively, | |
119 | have an effect on the flow or flows matched by the flow_mod. | |
12442ec5 | 120 | |
542cc9bb TG |
121 | - "updates idle timer": Whether the flow_mod resets the per-flow |
122 | timer that measures how long a flow has been idle. | |
12442ec5 | 123 | |
542cc9bb TG |
124 | - "updates hard timer": Whether the flow_mod resets the per-flow |
125 | timer that measures how long it has been since a flow was | |
126 | modified. | |
12442ec5 | 127 | |
542cc9bb TG |
128 | - "zeros counters": Whether the flow_mod resets per-flow packet |
129 | and byte counters to zero. | |
12442ec5 | 130 | |
542cc9bb TG |
131 | - "may add a new flow": Whether the flow_mod may add a new flow to |
132 | the flow table. (Obviously this is always true for "add" | |
133 | commands but in some OpenFlow versions "modify" and | |
134 | "modify-strict" can also add new flows.) | |
3432cb4e | 135 | |
542cc9bb TG |
136 | - "sends flow_removed message": Whether the flow_mod generates a |
137 | flow_removed message for the flow or flows that it affects. | |
12442ec5 BP |
138 | |
139 | An entry labeled "yes" means that the flow mod type does have the | |
140 | indicated behavior, "---" means that it does not, an empty cell means | |
141 | that the property is not applicable, and other values are explained | |
142 | below the table. | |
143 | ||
3432cb4e BP |
144 | OpenFlow 1.0 |
145 | ------------ | |
146 | ||
542cc9bb | 147 | ``` |
12442ec5 BP |
148 | MODIFY DELETE |
149 | ADD MODIFY STRICT DELETE STRICT | |
150 | === ====== ====== ====== ====== | |
3432cb4e | 151 | match on priority yes --- yes --- yes |
906087ee | 152 | match on out_port --- --- --- yes yes |
3432cb4e BP |
153 | match on flow_cookie --- --- --- --- --- |
154 | match on table_id --- --- --- --- --- | |
155 | controller chooses table_id --- --- --- | |
12442ec5 BP |
156 | updates flow_cookie yes yes yes |
157 | updates OFPFF_SEND_FLOW_REM yes + + | |
158 | honors OFPFF_CHECK_OVERLAP yes + + | |
159 | updates idle_timeout yes + + | |
160 | updates hard_timeout yes + + | |
161 | resets idle timer yes + + | |
162 | resets hard timer yes yes yes | |
163 | zeros counters yes + + | |
3432cb4e BP |
164 | may add a new flow yes yes yes |
165 | sends flow_removed message --- --- --- % % | |
166 | ||
167 | (+) "modify" and "modify-strict" only take these actions when they | |
168 | create a new flow, not when they update an existing flow. | |
169 | ||
170 | (%) "delete" and "delete_strict" generates a flow_removed message if | |
171 | the deleted flow or flows have the OFPFF_SEND_FLOW_REM flag set. | |
172 | (Each controller can separately control whether it wants to | |
173 | receive the generated messages.) | |
542cc9bb | 174 | ``` |
3432cb4e BP |
175 | |
176 | OpenFlow 1.1 | |
177 | ------------ | |
178 | ||
179 | OpenFlow 1.1 makes these changes: | |
180 | ||
542cc9bb TG |
181 | - The controller now must specify the table_id of the flow match |
182 | searched and into which a flow may be inserted. Behavior for a | |
183 | table_id of 255 is undefined. | |
3432cb4e | 184 | |
542cc9bb | 185 | - A flow_mod, except an "add", can now match on the flow_cookie. |
3432cb4e | 186 | |
542cc9bb TG |
187 | - When a flow_mod matches on the flow_cookie, "modify" and |
188 | "modify-strict" never insert a new flow. | |
3432cb4e | 189 | |
542cc9bb | 190 | ``` |
3432cb4e BP |
191 | MODIFY DELETE |
192 | ADD MODIFY STRICT DELETE STRICT | |
193 | === ====== ====== ====== ====== | |
194 | match on priority yes --- yes --- yes | |
195 | match on out_port --- --- --- yes yes | |
196 | match on flow_cookie --- yes yes yes yes | |
197 | match on table_id yes yes yes yes yes | |
198 | controller chooses table_id yes yes yes | |
199 | updates flow_cookie yes --- --- | |
200 | updates OFPFF_SEND_FLOW_REM yes + + | |
201 | honors OFPFF_CHECK_OVERLAP yes + + | |
202 | updates idle_timeout yes + + | |
203 | updates hard_timeout yes + + | |
204 | resets idle timer yes + + | |
205 | resets hard timer yes yes yes | |
206 | zeros counters yes + + | |
207 | may add a new flow yes # # | |
12442ec5 BP |
208 | sends flow_removed message --- --- --- % % |
209 | ||
210 | (+) "modify" and "modify-strict" only take these actions when they | |
211 | create a new flow, not when they update an existing flow. | |
212 | ||
213 | (%) "delete" and "delete_strict" generates a flow_removed message if | |
214 | the deleted flow or flows have the OFPFF_SEND_FLOW_REM flag set. | |
215 | (Each controller can separately control whether it wants to | |
216 | receive the generated messages.) | |
217 | ||
3432cb4e BP |
218 | (#) "modify" and "modify-strict" only add a new flow if the flow_mod |
219 | does not match on any bits of the flow cookie | |
542cc9bb | 220 | ``` |
3432cb4e BP |
221 | |
222 | OpenFlow 1.2 | |
223 | ------------ | |
224 | ||
225 | OpenFlow 1.2 makes these changes: | |
226 | ||
542cc9bb TG |
227 | - Only "add" commands ever add flows, "modify" and "modify-strict" |
228 | never do. | |
3432cb4e | 229 | |
542cc9bb TG |
230 | - A new flag OFPFF_RESET_COUNTS now controls whether "modify" and |
231 | "modify-strict" reset counters, whereas previously they never | |
232 | reset counters (except when they inserted a new flow). | |
3432cb4e | 233 | |
542cc9bb | 234 | ``` |
3432cb4e BP |
235 | MODIFY DELETE |
236 | ADD MODIFY STRICT DELETE STRICT | |
237 | === ====== ====== ====== ====== | |
238 | match on priority yes --- yes --- yes | |
239 | match on out_port --- --- --- yes yes | |
240 | match on flow_cookie --- yes yes yes yes | |
241 | match on table_id yes yes yes yes yes | |
242 | controller chooses table_id yes yes yes | |
243 | updates flow_cookie yes --- --- | |
244 | updates OFPFF_SEND_FLOW_REM yes --- --- | |
245 | honors OFPFF_CHECK_OVERLAP yes --- --- | |
246 | updates idle_timeout yes --- --- | |
247 | updates hard_timeout yes --- --- | |
248 | resets idle timer yes --- --- | |
249 | resets hard timer yes yes yes | |
250 | zeros counters yes & & | |
251 | may add a new flow yes --- --- | |
252 | sends flow_removed message --- --- --- % % | |
253 | ||
254 | (%) "delete" and "delete_strict" generates a flow_removed message if | |
255 | the deleted flow or flows have the OFPFF_SEND_FLOW_REM flag set. | |
256 | (Each controller can separately control whether it wants to | |
257 | receive the generated messages.) | |
258 | ||
259 | (&) "modify" and "modify-strict" reset counters if the | |
260 | OFPFF_RESET_COUNTS flag is specified. | |
542cc9bb | 261 | ``` |
3432cb4e BP |
262 | |
263 | OpenFlow 1.3 | |
264 | ------------ | |
265 | ||
266 | OpenFlow 1.3 makes these changes: | |
267 | ||
542cc9bb TG |
268 | - Behavior for a table_id of 255 is now defined, for "delete" and |
269 | "delete-strict" commands, as meaning to delete from all tables. | |
270 | A table_id of 255 is now explicitly invalid for other commands. | |
3432cb4e | 271 | |
542cc9bb TG |
272 | - New flags OFPFF_NO_PKT_COUNTS and OFPFF_NO_BYT_COUNTS for "add" |
273 | operations. | |
3432cb4e BP |
274 | |
275 | The table for 1.3 is the same as the one shown above for 1.2. | |
276 | ||
12442ec5 | 277 | |
c37c0382 AC |
278 | OpenFlow 1.4 |
279 | ------------ | |
280 | ||
ca26eb44 RB |
281 | OpenFlow 1.4 adds the "importance" field to flow_mods, but it does not |
282 | explicitly specify which kinds of flow_mods set the importance.For | |
283 | consistency, Open vSwitch uses the same rule for importance as for | |
284 | idle_timeout and hard_timeout, that is, only an "ADD" flow_mod sets | |
285 | the importance. (This issue has been filed with the ONF as EXT-496.) | |
c37c0382 | 286 | |
4d197ebb BP |
287 | OFPT_PACKET_IN |
288 | ============== | |
289 | ||
290 | The OpenFlow 1.1 specification for OFPT_PACKET_IN is confusing. The | |
291 | definition in OF1.1 openflow.h is[*]: | |
292 | ||
542cc9bb | 293 | ``` |
4d197ebb BP |
294 | /* Packet received on port (datapath -> controller). */ |
295 | struct ofp_packet_in { | |
296 | struct ofp_header header; | |
297 | uint32_t buffer_id; /* ID assigned by datapath. */ | |
298 | uint32_t in_port; /* Port on which frame was received. */ | |
299 | uint32_t in_phy_port; /* Physical Port on which frame was received. */ | |
300 | uint16_t total_len; /* Full length of frame. */ | |
301 | uint8_t reason; /* Reason packet is being sent (one of OFPR_*) */ | |
302 | uint8_t table_id; /* ID of the table that was looked up */ | |
303 | uint8_t data[0]; /* Ethernet frame, halfway through 32-bit word, | |
304 | so the IP header is 32-bit aligned. The | |
305 | amount of data is inferred from the length | |
306 | field in the header. Because of padding, | |
307 | offsetof(struct ofp_packet_in, data) == | |
308 | sizeof(struct ofp_packet_in) - 2. */ | |
309 | }; | |
310 | OFP_ASSERT(sizeof(struct ofp_packet_in) == 24); | |
542cc9bb | 311 | ``` |
4d197ebb BP |
312 | |
313 | The confusing part is the comment on the data[] member. This comment | |
314 | is a leftover from OF1.0 openflow.h, in which the comment was correct: | |
315 | sizeof(struct ofp_packet_in) is 20 in OF1.0 and offsetof(struct | |
316 | ofp_packet_in, data) is 18. When OF1.1 was written, the structure | |
317 | members were changed but the comment was carelessly not updated, and | |
318 | the comment became wrong: sizeof(struct ofp_packet_in) and | |
319 | offsetof(struct ofp_packet_in, data) are both 24 in OF1.1. | |
320 | ||
321 | That leaves the question of how to implement ofp_packet_in in OF1.1. | |
322 | The OpenFlow reference implementation for OF1.1 does not include any | |
323 | padding, that is, the first byte of the encapsulated frame immediately | |
324 | follows the 'table_id' member without a gap. Open vSwitch therefore | |
325 | implements it the same way for compatibility. | |
326 | ||
327 | For an earlier discussion, please see the thread archived at: | |
328 | https://mailman.stanford.edu/pipermail/openflow-discuss/2011-August/002604.html | |
329 | ||
330 | [*] The quoted definition is directly from OF1.1. Definitions used | |
331 | inside OVS omit the 8-byte ofp_header members, so the sizes in | |
332 | this discussion are 8 bytes larger than those declared in OVS | |
333 | header files. | |
334 | ||
335 | ||
df778240 BP |
336 | VLAN Matching |
337 | ============= | |
338 | ||
339 | The 802.1Q VLAN header causes more trouble than any other 4 bytes in | |
340 | networking. More specifically, three versions of OpenFlow and Open | |
341 | vSwitch have among them four different ways to match the contents and | |
342 | presence of the VLAN header. The following table describes how each | |
343 | version works. | |
344 | ||
345 | Match NXM OF1.0 OF1.1 OF1.2 | |
346 | ----- --------- ----------- ----------- ------------ | |
347 | [1] 0000/0000 ????/1,??/? ????/1,??/? 0000/0000,-- | |
348 | [2] 0000/ffff ffff/0,??/? ffff/0,??/? 0000/ffff,-- | |
349 | [3] 1xxx/1fff 0xxx/0,??/1 0xxx/0,??/1 1xxx/ffff,-- | |
350 | [4] z000/f000 ????/1,0y/0 fffe/0,0y/0 1000/1000,0y | |
351 | [5] zxxx/ffff 0xxx/0,0y/0 0xxx/0,0y/0 1xxx/ffff,0y | |
352 | [6] 0000/0fff <none> <none> <none> | |
353 | [7] 0000/f000 <none> <none> <none> | |
354 | [8] 0000/efff <none> <none> <none> | |
355 | [9] 1001/1001 <none> <none> 1001/1001,-- | |
356 | [10] 3000/3000 <none> <none> <none> | |
357 | ||
358 | Each column is interpreted as follows. | |
359 | ||
542cc9bb | 360 | - Match: See the list below. |
df778240 | 361 | |
542cc9bb TG |
362 | - NXM: xxxx/yyyy means NXM_OF_VLAN_TCI_W with value xxxx and mask |
363 | yyyy. A mask of 0000 is equivalent to omitting | |
364 | NXM_OF_VLAN_TCI(_W), a mask of ffff is equivalent to | |
365 | NXM_OF_VLAN_TCI. | |
df778240 | 366 | |
542cc9bb TG |
367 | - OF1.0 and OF1.1: wwww/x,yy/z means dl_vlan wwww, OFPFW_DL_VLAN |
368 | x, dl_vlan_pcp yy, and OFPFW_DL_VLAN_PCP z. ? means that the | |
369 | given nibble is ignored (and conventionally 0 for wwww or yy, | |
370 | conventionally 1 for x or z). <none> means that the given match | |
371 | is not supported. | |
df778240 | 372 | |
542cc9bb TG |
373 | - OF1.2: xxxx/yyyy,zz means OXM_OF_VLAN_VID_W with value xxxx and |
374 | mask yyyy, and OXM_OF_VLAN_PCP (which is not maskable) with | |
375 | value zz. A mask of 0000 is equivalent to omitting | |
376 | OXM_OF_VLAN_VID(_W), a mask of ffff is equivalent to | |
377 | OXM_OF_VLAN_VID. -- means that OXM_OF_VLAN_PCP is omitted. | |
378 | <none> means that the given match is not supported. | |
df778240 BP |
379 | |
380 | The matches are: | |
381 | ||
382 | [1] Matches any packet, that is, one without an 802.1Q header or with | |
383 | an 802.1Q header with any TCI value. | |
384 | ||
385 | [2] Matches only packets without an 802.1Q header. | |
386 | ||
387 | NXM: Any match with (vlan_tci == 0) and (vlan_tci_mask & 0x1000) | |
388 | != 0 is equivalent to the one listed in the table. | |
389 | ||
390 | OF1.0: The spec doesn't define behavior if dl_vlan is set to | |
391 | 0xffff and OFPFW_DL_VLAN_PCP is not set. | |
392 | ||
393 | OF1.1: The spec says explicitly to ignore dl_vlan_pcp when | |
394 | dl_vlan is set to 0xffff. | |
395 | ||
396 | OF1.2: The spec doesn't say what should happen if (vlan_vid == 0) | |
397 | and (vlan_vid_mask & 0x1000) != 0 but (vlan_vid_mask != 0x1000), | |
398 | but it would be straightforward to also interpret as [2]. | |
399 | ||
400 | [3] Matches only packets that have an 802.1Q header with VID xxx (and | |
401 | any PCP). | |
402 | ||
403 | [4] Matches only packets that have an 802.1Q header with PCP y (and | |
404 | any VID). | |
405 | ||
406 | NXM: z is ((y << 1) | 1). | |
407 | ||
408 | OF1.0: The spec isn't very clear, but OVS implements it this way. | |
409 | ||
410 | OF1.2: Presumably other masks such that (vlan_vid_mask & 0x1fff) | |
411 | == 0x1000 would also work, but the spec doesn't define their | |
412 | behavior. | |
413 | ||
414 | [5] Matches only packets that have an 802.1Q header with VID xxx and | |
415 | PCP y. | |
416 | ||
417 | NXM: z is ((y << 1) | 1). | |
418 | ||
419 | OF1.2: Presumably other masks such that (vlan_vid_mask & 0x1fff) | |
420 | == 0x1fff would also work. | |
421 | ||
422 | [6] Matches packets with no 802.1Q header or with an 802.1Q header | |
423 | with a VID of 0. Only possible with NXM. | |
424 | ||
425 | [7] Matches packets with no 802.1Q header or with an 802.1Q header | |
426 | with a PCP of 0. Only possible with NXM. | |
427 | ||
428 | [8] Matches packets with no 802.1Q header or with an 802.1Q header | |
429 | with both VID and PCP of 0. Only possible with NXM. | |
430 | ||
431 | [9] Matches only packets that have an 802.1Q header with an | |
432 | odd-numbered VID (and any PCP). Only possible with NXM and | |
433 | OF1.2. (This is just an example; one can match on any desired | |
434 | VID bit pattern.) | |
435 | ||
436 | [10] Matches only packets that have an 802.1Q header with an | |
437 | odd-numbered PCP (and any VID). Only possible with NXM. (This | |
438 | is just an example; one can match on any desired VID bit | |
439 | pattern.) | |
440 | ||
441 | Additional notes: | |
442 | ||
542cc9bb TG |
443 | - OF1.2: The top three bits of OXM_OF_VLAN_VID are fixed to zero, |
444 | so bits 13, 14, and 15 in the masks listed in the table may be | |
445 | set to arbitrary values, as long as the corresponding value bits | |
446 | are also zero. The suggested ffff mask for [2], [3], and [5] | |
447 | allows a shorter OXM representation (the mask is omitted) than | |
448 | the minimal 1fff mask. | |
df778240 BP |
449 | |
450 | ||
f66b87de BP |
451 | Flow Cookies |
452 | ============ | |
453 | ||
454 | OpenFlow 1.0 and later versions have the concept of a "flow cookie", | |
455 | which is a 64-bit integer value attached to each flow. The treatment | |
456 | of the flow cookie has varied greatly across OpenFlow versions, | |
457 | however. | |
458 | ||
459 | In OpenFlow 1.0: | |
460 | ||
542cc9bb | 461 | - OFPFC_ADD set the cookie in the flow that it added. |
f66b87de | 462 | |
542cc9bb TG |
463 | - OFPFC_MODIFY and OFPFC_MODIFY_STRICT updated the cookie for |
464 | the flow or flows that it modified. | |
f66b87de | 465 | |
542cc9bb | 466 | - OFPST_FLOW messages included the flow cookie. |
f66b87de | 467 | |
542cc9bb TG |
468 | - OFPT_FLOW_REMOVED messages reported the cookie of the flow |
469 | that was removed. | |
f66b87de BP |
470 | |
471 | OpenFlow 1.1 made the following changes: | |
472 | ||
542cc9bb TG |
473 | - Flow mod operations OFPFC_MODIFY, OFPFC_MODIFY_STRICT, |
474 | OFPFC_DELETE, and OFPFC_DELETE_STRICT, plus flow stats | |
475 | requests and aggregate stats requests, gained the ability to | |
476 | match on flow cookies with an arbitrary mask. | |
f66b87de | 477 | |
542cc9bb TG |
478 | - OFPFC_MODIFY and OFPFC_MODIFY_STRICT were changed to add a |
479 | new flow, in the case of no match, only if the flow table | |
480 | modification operation did not match on the cookie field. | |
481 | (In OpenFlow 1.0, modify operations always added a new flow | |
482 | when there was no match.) | |
f66b87de | 483 | |
542cc9bb TG |
484 | - OFPFC_MODIFY and OFPFC_MODIFY_STRICT no longer updated flow |
485 | cookies. | |
f66b87de BP |
486 | |
487 | OpenFlow 1.2 made the following changes: | |
488 | ||
542cc9bb TG |
489 | - OFPC_MODIFY and OFPFC_MODIFY_STRICT were changed to never |
490 | add a new flow, regardless of whether the flow cookie was | |
491 | used for matching. | |
f66b87de BP |
492 | |
493 | Open vSwitch support for OpenFlow 1.0 implements the OpenFlow 1.0 | |
494 | behavior with the following extensions: | |
495 | ||
542cc9bb TG |
496 | - An NXM extension field NXM_NX_COOKIE(_W) allows the NXM |
497 | versions of OFPFC_MODIFY, OFPFC_MODIFY_STRICT, OFPFC_DELETE, | |
498 | and OFPFC_DELETE_STRICT flow_mods, plus flow stats requests | |
499 | and aggregate stats requests, to match on flow cookies with | |
500 | arbitrary masks. This is much like the equivalent OpenFlow | |
501 | 1.1 feature. | |
502 | ||
503 | - Like OpenFlow 1.1, OFPC_MODIFY and OFPFC_MODIFY_STRICT add a | |
504 | new flow if there is no match and the mask is zero (or not | |
505 | given). | |
506 | ||
507 | - The "cookie" field in OFPT_FLOW_MOD and NXT_FLOW_MOD messages | |
508 | is used as the cookie value for OFPFC_ADD commands, as | |
509 | described in OpenFlow 1.0. For OFPFC_MODIFY and | |
510 | OFPFC_MODIFY_STRICT commands, the "cookie" field is used as a | |
511 | new cookie for flows that match unless it is UINT64_MAX, in | |
512 | which case the flow's cookie is not updated. | |
513 | ||
514 | - NXT_PACKET_IN (the Nicira extended version of | |
515 | OFPT_PACKET_IN) reports the cookie of the rule that | |
516 | generated the packet, or all-1-bits if no rule generated the | |
517 | packet. (Older versions of OVS used all-0-bits instead of | |
518 | all-1-bits.) | |
f66b87de | 519 | |
623e1caf JP |
520 | The following table shows the handling of different protocols when |
521 | receiving OFPFC_MODIFY and OFPFC_MODIFY_STRICT messages. A mask of 0 | |
522 | indicates either an explicit mask of zero or an implicit one by not | |
523 | specifying the NXM_NX_COOKIE(_W) field. | |
524 | ||
542cc9bb | 525 | ``` |
623e1caf JP |
526 | Match Update Add on miss Add on miss |
527 | cookie cookie mask!=0 mask==0 | |
528 | ====== ====== =========== =========== | |
529 | OpenFlow 1.0 no yes <always add on miss> | |
530 | OpenFlow 1.1 yes no no yes | |
531 | OpenFlow 1.2 yes no no no | |
532 | NXM yes yes* no yes | |
533 | ||
534 | * Updates the flow's cookie unless the "cookie" field is UINT64_MAX. | |
542cc9bb | 535 | ``` |
f66b87de | 536 | |
66abb12b BP |
537 | Multiple Table Support |
538 | ====================== | |
539 | ||
540 | OpenFlow 1.0 has only rudimentary support for multiple flow tables. | |
541 | Notably, OpenFlow 1.0 does not allow the controller to specify the | |
542 | flow table to which a flow is to be added. Open vSwitch adds an | |
543 | extension for this purpose, which is enabled on a per-OpenFlow | |
544 | connection basis using the NXT_FLOW_MOD_TABLE_ID message. When the | |
545 | extension is enabled, the upper 8 bits of the 'command' member in an | |
546 | OFPT_FLOW_MOD or NXT_FLOW_MOD message designates the table to which a | |
547 | flow is to be added. | |
548 | ||
549 | The Open vSwitch software switch implementation offers 255 flow | |
550 | tables. On packet ingress, only the first flow table (table 0) is | |
551 | searched, and the contents of the remaining tables are not considered | |
552 | in any way. Tables other than table 0 only come into play when an | |
553 | NXAST_RESUBMIT_TABLE action specifies another table to search. | |
554 | ||
555 | Tables 128 and above are reserved for use by the switch itself. | |
556 | Controllers should use only tables 0 through 127. | |
557 | ||
558 | ||
d31f1109 JP |
559 | IPv6 |
560 | ==== | |
561 | ||
562 | Open vSwitch supports stateless handling of IPv6 packets. Flows can be | |
563 | written to support matching TCP, UDP, and ICMPv6 headers within an IPv6 | |
685a51a5 JP |
564 | packet. Deeper matching of some Neighbor Discovery messages is also |
565 | supported. | |
d31f1109 JP |
566 | |
567 | IPv6 was not designed to interact well with middle-boxes. This, | |
568 | combined with Open vSwitch's stateless nature, have affected the | |
569 | processing of IPv6 traffic, which is detailed below. | |
570 | ||
571 | Extension Headers | |
572 | ----------------- | |
573 | ||
574 | The base IPv6 header is incredibly simple with the intention of only | |
575 | containing information relevant for routing packets between two | |
576 | endpoints. IPv6 relies heavily on the use of extension headers to | |
577 | provide any other functionality. Unfortunately, the extension headers | |
578 | were designed in such a way that it is impossible to move to the next | |
579 | header (including the layer-4 payload) unless the current header is | |
580 | understood. | |
581 | ||
582 | Open vSwitch will process the following extension headers and continue | |
583 | to the next header: | |
584 | ||
542cc9bb TG |
585 | * Fragment (see the next section) |
586 | * AH (Authentication Header) | |
587 | * Hop-by-Hop Options | |
588 | * Routing | |
589 | * Destination Options | |
d31f1109 JP |
590 | |
591 | When a header is encountered that is not in that list, it is considered | |
592 | "terminal". A terminal header's IPv6 protocol value is stored in | |
593 | "nw_proto" for matching purposes. If a terminal header is TCP, UDP, or | |
594 | ICMPv6, the packet will be further processed in an attempt to extract | |
595 | layer-4 information. | |
596 | ||
597 | Fragments | |
598 | --------- | |
599 | ||
600 | IPv6 requires that every link in the internet have an MTU of 1280 octets | |
601 | or greater (RFC 2460). As such, a terminal header (as described above in | |
602 | "Extension Headers") in the first fragment should generally be | |
603 | reachable. In this case, the terminal header's IPv6 protocol type is | |
604 | stored in the "nw_proto" field for matching purposes. If a terminal | |
605 | header cannot be found in the first fragment (one with a fragment offset | |
606 | of zero), the "nw_proto" field is set to 0. Subsequent fragments (those | |
607 | with a non-zero fragment offset) have the "nw_proto" field set to the | |
608 | IPv6 protocol type for fragments (44). | |
609 | ||
610 | Jumbograms | |
611 | ---------- | |
612 | ||
613 | An IPv6 jumbogram (RFC 2675) is a packet containing a payload longer | |
614 | than 65,535 octets. A jumbogram is only relevant in subnets with a link | |
615 | MTU greater than 65,575 octets, and are not required to be supported on | |
616 | nodes that do not connect to link with such large MTUs. Currently, Open | |
617 | vSwitch doesn't process jumbograms. | |
618 | ||
619 | ||
946350dc BP |
620 | In-Band Control |
621 | =============== | |
622 | ||
56e9c3b9 BP |
623 | Motivation |
624 | ---------- | |
625 | ||
626 | An OpenFlow switch must establish and maintain a TCP network | |
627 | connection to its controller. There are two basic ways to categorize | |
628 | the network that this connection traverses: either it is completely | |
629 | separate from the one that the switch is otherwise controlling, or its | |
630 | path may overlap the network that the switch controls. We call the | |
631 | former case "out-of-band control", the latter case "in-band control". | |
632 | ||
633 | Out-of-band control has the following benefits: | |
634 | ||
542cc9bb TG |
635 | - Simplicity: Out-of-band control slightly simplifies the switch |
636 | implementation. | |
56e9c3b9 | 637 | |
542cc9bb TG |
638 | - Reliability: Excessive switch traffic volume cannot interfere |
639 | with control traffic. | |
56e9c3b9 | 640 | |
542cc9bb TG |
641 | - Integrity: Machines not on the control network cannot |
642 | impersonate a switch or a controller. | |
56e9c3b9 | 643 | |
542cc9bb TG |
644 | - Confidentiality: Machines not on the control network cannot |
645 | snoop on control traffic. | |
56e9c3b9 BP |
646 | |
647 | In-band control, on the other hand, has the following advantages: | |
648 | ||
542cc9bb TG |
649 | - No dedicated port: There is no need to dedicate a physical |
650 | switch port to control, which is important on switches that have | |
651 | few ports (e.g. wireless routers, low-end embedded platforms). | |
56e9c3b9 | 652 | |
542cc9bb TG |
653 | - No dedicated network: There is no need to build and maintain a |
654 | separate control network. This is important in many | |
655 | environments because it reduces proliferation of switches and | |
656 | wiring. | |
56e9c3b9 BP |
657 | |
658 | Open vSwitch supports both out-of-band and in-band control. This | |
659 | section describes the principles behind in-band control. See the | |
660 | description of the Controller table in ovs-vswitchd.conf.db(5) to | |
661 | configure OVS for in-band control. | |
662 | ||
663 | Principles | |
664 | ---------- | |
665 | ||
666 | The fundamental principle of in-band control is that an OpenFlow | |
667 | switch must recognize and switch control traffic without involving the | |
668 | OpenFlow controller. All the details of implementing in-band control | |
669 | are special cases of this principle. | |
670 | ||
671 | The rationale for this principle is simple. If the switch does not | |
672 | handle in-band control traffic itself, then it will be caught in a | |
673 | contradiction: it must contact the controller, but it cannot, because | |
674 | only the controller can set up the flows that are needed to contact | |
675 | the controller. | |
676 | ||
677 | The following points describe important special cases of this | |
678 | principle. | |
679 | ||
542cc9bb TG |
680 | - In-band control must be implemented regardless of whether the |
681 | switch is connected. | |
682 | ||
683 | It is tempting to implement the in-band control rules only when | |
684 | the switch is not connected to the controller, using the | |
685 | reasoning that the controller should have complete control once | |
686 | it has established a connection with the switch. | |
687 | ||
688 | This does not work in practice. Consider the case where the | |
689 | switch is connected to the controller. Occasionally it can | |
690 | happen that the controller forgets or otherwise needs to obtain | |
691 | the MAC address of the switch. To do so, the controller sends a | |
692 | broadcast ARP request. A switch that implements the in-band | |
693 | control rules only when it is disconnected will then send an | |
694 | OFPT_PACKET_IN message up to the controller. The controller will | |
695 | be unable to respond, because it does not know the MAC address of | |
696 | the switch. This is a deadlock situation that can only be | |
697 | resolved by the switch noticing that its connection to the | |
698 | controller has hung and reconnecting. | |
699 | ||
700 | - In-band control must override flows set up by the controller. | |
701 | ||
702 | It is reasonable to assume that flows set up by the OpenFlow | |
703 | controller should take precedence over in-band control, on the | |
704 | basis that the controller should be in charge of the switch. | |
705 | ||
706 | Again, this does not work in practice. Reasonable controller | |
707 | implementations may set up a "last resort" fallback rule that | |
708 | wildcards every field and, e.g., sends it up to the controller or | |
709 | discards it. If a controller does that, then it will isolate | |
710 | itself from the switch. | |
711 | ||
712 | - The switch must recognize all control traffic. | |
713 | ||
714 | The fundamental principle of in-band control states, in part, | |
715 | that a switch must recognize control traffic without involving | |
716 | the OpenFlow controller. More specifically, the switch must | |
717 | recognize *all* control traffic. "False negatives", that is, | |
718 | packets that constitute control traffic but that the switch does | |
719 | not recognize as control traffic, lead to control traffic storms. | |
720 | ||
721 | Consider an OpenFlow switch that only recognizes control packets | |
722 | sent to or from that switch. Now suppose that two switches of | |
723 | this type, named A and B, are connected to ports on an Ethernet | |
724 | hub (not a switch) and that an OpenFlow controller is connected | |
725 | to a third hub port. In this setup, control traffic sent by | |
726 | switch A will be seen by switch B, which will send it to the | |
727 | controller as part of an OFPT_PACKET_IN message. Switch A will | |
728 | then see the OFPT_PACKET_IN message's packet, re-encapsulate it | |
729 | in another OFPT_PACKET_IN, and send it to the controller. Switch | |
730 | B will then see that OFPT_PACKET_IN, and so on in an infinite | |
731 | loop. | |
732 | ||
733 | Incidentally, the consequences of "false positives", where | |
734 | packets that are not control traffic are nevertheless recognized | |
735 | as control traffic, are much less severe. The controller will | |
736 | not be able to control their behavior, but the network will | |
737 | remain in working order. False positives do constitute a | |
738 | security problem. | |
739 | ||
740 | - The switch should use echo-requests to detect disconnection. | |
741 | ||
742 | TCP will notice that a connection has hung, but this can take a | |
743 | considerable amount of time. For example, with default settings | |
744 | the Linux kernel TCP implementation will retransmit for between | |
745 | 13 and 30 minutes, depending on the connection's retransmission | |
746 | timeout, according to kernel documentation. This is far too long | |
747 | for a switch to be disconnected, so an OpenFlow switch should | |
748 | implement its own connection timeout. OpenFlow OFPT_ECHO_REQUEST | |
749 | messages are the best way to do this, since they test the | |
750 | OpenFlow connection itself. | |
56e9c3b9 BP |
751 | |
752 | Implementation | |
753 | -------------- | |
754 | ||
755 | This section describes how Open vSwitch implements in-band control. | |
756 | Correctly implementing in-band control has proven difficult due to its | |
757 | many subtleties, and has thus gone through many iterations. Please | |
758 | read through and understand the reasoning behind the chosen rules | |
759 | before making modifications. | |
760 | ||
761 | Open vSwitch implements in-band control as "hidden" flows, that is, | |
762 | flows that are not visible through OpenFlow, and at a higher priority | |
763 | than wildcarded flows can be set up through OpenFlow. This is done so | |
764 | that the OpenFlow controller cannot interfere with them and possibly | |
765 | break connectivity with its switches. It is possible to see all | |
766 | flows, including in-band ones, with the ovs-appctl "bridge/dump-flows" | |
767 | command. | |
946350dc BP |
768 | |
769 | The Open vSwitch implementation of in-band control can hide traffic to | |
770 | arbitrary "remotes", where each remote is one TCP port on one IP address. | |
771 | Currently the remotes are automatically configured as the in-band OpenFlow | |
772 | controllers plus the OVSDB managers, if any. (The latter is a requirement | |
773 | because OVSDB managers are responsible for configuring OpenFlow controllers, | |
774 | so if the manager cannot be reached then OpenFlow cannot be reconfigured.) | |
775 | ||
776 | The following rules (with the OFPP_NORMAL action) are set up on any bridge | |
777 | that has any remotes: | |
778 | ||
779 | (a) DHCP requests sent from the local port. | |
780 | (b) ARP replies to the local port's MAC address. | |
781 | (c) ARP requests from the local port's MAC address. | |
782 | ||
783 | In-band also sets up the following rules for each unique next-hop MAC | |
784 | address for the remotes' IPs (the "next hop" is either the remote | |
785 | itself, if it is on a local subnet, or the gateway to reach the remote): | |
786 | ||
787 | (d) ARP replies to the next hop's MAC address. | |
788 | (e) ARP requests from the next hop's MAC address. | |
789 | ||
790 | In-band also sets up the following rules for each unique remote IP address: | |
791 | ||
792 | (f) ARP replies containing the remote's IP address as a target. | |
793 | (g) ARP requests containing the remote's IP address as a source. | |
794 | ||
795 | In-band also sets up the following rules for each unique remote (IP,port) | |
796 | pair: | |
797 | ||
798 | (h) TCP traffic to the remote's IP and port. | |
799 | (i) TCP traffic from the remote's IP and port. | |
800 | ||
801 | The goal of these rules is to be as narrow as possible to allow a | |
802 | switch to join a network and be able to communicate with the | |
803 | remotes. As mentioned earlier, these rules have higher priority | |
804 | than the controller's rules, so if they are too broad, they may | |
805 | prevent the controller from implementing its policy. As such, | |
806 | in-band actively monitors some aspects of flow and packet processing | |
807 | so that the rules can be made more precise. | |
808 | ||
809 | In-band control monitors attempts to add flows into the datapath that | |
810 | could interfere with its duties. The datapath only allows exact | |
811 | match entries, so in-band control is able to be very precise about | |
812 | the flows it prevents. Flows that miss in the datapath are sent to | |
813 | userspace to be processed, so preventing these flows from being | |
814 | cached in the "fast path" does not affect correctness. The only type | |
815 | of flow that is currently prevented is one that would prevent DHCP | |
816 | replies from being seen by the local port. For example, a rule that | |
817 | forwarded all DHCP traffic to the controller would not be allowed, | |
818 | but one that forwarded to all ports (including the local port) would. | |
819 | ||
820 | As mentioned earlier, packets that miss in the datapath are sent to | |
821 | the userspace for processing. The userspace has its own flow table, | |
822 | the "classifier", so in-band checks whether any special processing | |
823 | is needed before the classifier is consulted. If a packet is a DHCP | |
824 | response to a request from the local port, the packet is forwarded to | |
825 | the local port, regardless of the flow table. Note that this requires | |
826 | L7 processing of DHCP replies to determine whether the 'chaddr' field | |
827 | matches the MAC address of the local port. | |
828 | ||
829 | It is interesting to note that for an L3-based in-band control | |
830 | mechanism, the majority of rules are devoted to ARP traffic. At first | |
831 | glance, some of these rules appear redundant. However, each serves an | |
832 | important role. First, in order to determine the MAC address of the | |
833 | remote side (controller or gateway) for other ARP rules, we must allow | |
834 | ARP traffic for our local port with rules (b) and (c). If we are | |
835 | between a switch and its connection to the remote, we have to | |
836 | allow the other switch's ARP traffic to through. This is done with | |
837 | rules (d) and (e), since we do not know the addresses of the other | |
838 | switches a priori, but do know the remote's or gateway's. Finally, | |
839 | if the remote is running in a local guest VM that is not reached | |
840 | through the local port, the switch that is connected to the VM must | |
841 | allow ARP traffic based on the remote's IP address, since it will | |
842 | not know the MAC address of the local port that is sending the traffic | |
843 | or the MAC address of the remote in the guest VM. | |
844 | ||
845 | With a few notable exceptions below, in-band should work in most | |
846 | network setups. The following are considered "supported' in the | |
847 | current implementation: | |
848 | ||
542cc9bb TG |
849 | - Locally Connected. The switch and remote are on the same |
850 | subnet. This uses rules (a), (b), (c), (h), and (i). | |
851 | ||
852 | - Reached through Gateway. The switch and remote are on | |
853 | different subnets and must go through a gateway. This uses | |
854 | rules (a), (b), (c), (h), and (i). | |
855 | ||
856 | - Between Switch and Remote. This switch is between another | |
857 | switch and the remote, and we want to allow the other | |
858 | switch's traffic through. This uses rules (d), (e), (h), and | |
859 | (i). It uses (b) and (c) indirectly in order to know the MAC | |
860 | address for rules (d) and (e). Note that DHCP for the other | |
861 | switch will not work unless an OpenFlow controller explicitly lets this | |
862 | switch pass the traffic. | |
863 | ||
864 | - Between Switch and Gateway. This switch is between another | |
865 | switch and the gateway, and we want to allow the other switch's | |
866 | traffic through. This uses the same rules and logic as the | |
867 | "Between Switch and Remote" configuration described earlier. | |
868 | ||
869 | - Remote on Local VM. The remote is a guest VM on the | |
870 | system running in-band control. This uses rules (a), (b), (c), | |
871 | (h), and (i). | |
872 | ||
873 | - Remote on Local VM with Different Networks. The remote | |
874 | is a guest VM on the system running in-band control, but the | |
875 | local port is not used to connect to the remote. For | |
876 | example, an IP address is configured on eth0 of the switch. The | |
877 | remote's VM is connected through eth1 of the switch, but an | |
878 | IP address has not been configured for that port on the switch. | |
879 | As such, the switch will use eth0 to connect to the remote, | |
880 | and eth1's rules about the local port will not work. In the | |
881 | example, the switch attached to eth0 would use rules (a), (b), | |
882 | (c), (h), and (i) on eth0. The switch attached to eth1 would use | |
883 | rules (f), (g), (h), and (i). | |
946350dc BP |
884 | |
885 | The following are explicitly *not* supported by in-band control: | |
886 | ||
542cc9bb TG |
887 | - Specify Remote by Name. Currently, the remote must be |
888 | identified by IP address. A naive approach would be to permit | |
889 | all DNS traffic. Unfortunately, this would prevent the | |
890 | controller from defining any policy over DNS. Since switches | |
891 | that are located behind us need to connect to the remote, | |
892 | in-band cannot simply add a rule that allows DNS traffic from | |
893 | the local port. The "correct" way to support this is to parse | |
894 | DNS requests to allow all traffic related to a request for the | |
895 | remote's name through. Due to the potential security | |
896 | problems and amount of processing, we decided to hold off for | |
897 | the time-being. | |
898 | ||
899 | - Differing Remotes for Switches. All switches must know | |
900 | the L3 addresses for all the remotes that other switches | |
901 | may use, since rules need to be set up to allow traffic related | |
902 | to those remotes through. See rules (f), (g), (h), and (i). | |
903 | ||
904 | - Differing Routes for Switches. In order for the switch to | |
905 | allow other switches to connect to a remote through a | |
906 | gateway, it allows the gateway's traffic through with rules (d) | |
907 | and (e). If the routes to the remote differ for the two | |
908 | switches, we will not know the MAC address of the alternate | |
909 | gateway. | |
946350dc BP |
910 | |
911 | ||
f25d0cf3 BP |
912 | Action Reproduction |
913 | =================== | |
914 | ||
915 | It seems likely that many controllers, at least at startup, use the | |
916 | OpenFlow "flow statistics" request to obtain existing flows, then | |
917 | compare the flows' actions against the actions that they expect to | |
918 | find. Before version 1.8.0, Open vSwitch always returned exact, | |
919 | byte-for-byte copies of the actions that had been added to the flow | |
920 | table. The current version of Open vSwitch does not always do this in | |
921 | some exceptional cases. This section lists the exceptions that | |
922 | controller authors must keep in mind if they compare actual actions | |
923 | against desired actions in a bytewise fashion: | |
924 | ||
542cc9bb TG |
925 | - Open vSwitch zeros padding bytes in action structures, |
926 | regardless of their values when the flows were added. | |
f25d0cf3 | 927 | |
542cc9bb TG |
928 | - Open vSwitch "normalizes" the instructions in OpenFlow 1.1 |
929 | (and later) in the following way: | |
d01c980f | 930 | |
542cc9bb TG |
931 | * OVS sorts the instructions into the following order: |
932 | Apply-Actions, Clear-Actions, Write-Actions, | |
933 | Write-Metadata, Goto-Table. | |
d01c980f | 934 | |
542cc9bb TG |
935 | * OVS drops Apply-Actions instructions that have empty |
936 | action lists. | |
d01c980f | 937 | |
542cc9bb TG |
938 | * OVS drops Write-Actions instructions that have empty |
939 | action sets. | |
d01c980f | 940 | |
f25d0cf3 BP |
941 | Please report other discrepancies, if you notice any, so that we can |
942 | fix or document them. | |
943 | ||
944 | ||
d31f1109 JP |
945 | Suggestions |
946 | =========== | |
947 | ||
948 | Suggestions to improve Open vSwitch are welcome at discuss@openvswitch.org. |