]>
Commit | Line | Data |
---|---|---|
542cc9bb TG |
1 | Design Decisions In Open vSwitch |
2 | ================================ | |
d31f1109 JP |
3 | |
4 | This document describes design decisions that went into implementing | |
5 | Open vSwitch. While we believe these to be reasonable decisions, it is | |
6 | impossible to predict how Open vSwitch will be used in all environments. | |
7 | Understanding assumptions made by Open vSwitch is critical to a | |
8 | successful deployment. The end of this document contains contact | |
9 | information that can be used to let us know how we can make Open vSwitch | |
10 | more generally useful. | |
11 | ||
80d5aefd BP |
12 | Asynchronous Messages |
13 | ===================== | |
14 | ||
15 | Over time, Open vSwitch has added many knobs that control whether a | |
16 | given controller receives OpenFlow asynchronous messages. This | |
17 | section describes how all of these features interact. | |
18 | ||
19 | First, a service controller never receives any asynchronous messages | |
4550b647 MM |
20 | unless it changes its miss_send_len from the service controller |
21 | default of zero in one of the following ways: | |
22 | ||
542cc9bb | 23 | - Sending an OFPT_SET_CONFIG message with nonzero miss_send_len. |
4550b647 | 24 | |
542cc9bb TG |
25 | - Sending any NXT_SET_ASYNC_CONFIG message: as a side effect, this |
26 | message changes the miss_send_len to | |
27 | OFP_DEFAULT_MISS_SEND_LEN (128) for service controllers. | |
80d5aefd BP |
28 | |
29 | Second, OFPT_FLOW_REMOVED and NXT_FLOW_REMOVED messages are generated | |
30 | only if the flow that was removed had the OFPFF_SEND_FLOW_REM flag | |
31 | set. | |
32 | ||
a7349929 BP |
33 | Third, OFPT_PACKET_IN and NXT_PACKET_IN messages are sent only to |
34 | OpenFlow controller connections that have the correct connection ID | |
35 | (see "struct nx_controller_id" and "struct nx_action_controller"): | |
36 | ||
542cc9bb TG |
37 | - For packet-in messages generated by a NXAST_CONTROLLER action, |
38 | the controller ID specified in the action. | |
a7349929 | 39 | |
542cc9bb TG |
40 | - For other packet-in messages, controller ID zero. (This is the |
41 | default ID when an OpenFlow controller does not configure one.) | |
a7349929 | 42 | |
80d5aefd BP |
43 | Finally, Open vSwitch consults a per-connection table indexed by the |
44 | message type, reason code, and current role. The following table | |
45 | shows how this table is initialized by default when an OpenFlow | |
46 | connection is made. An entry labeled "yes" means that the message is | |
47 | sent, an entry labeled "---" means that the message is suppressed. | |
48 | ||
542cc9bb | 49 | ``` |
80d5aefd BP |
50 | master/ |
51 | message and reason code other slave | |
52 | ---------------------------------------- ------- ----- | |
53 | OFPT_PACKET_IN / NXT_PACKET_IN | |
54 | OFPR_NO_MATCH yes --- | |
55 | OFPR_ACTION yes --- | |
56 | OFPR_INVALID_TTL --- --- | |
029ca940 | 57 | OFPR_ACTION_SET (OF1.4+) yes --- |
3a11fd5b | 58 | OFPR_GROUP (OF1.4+) yes --- |
80d5aefd BP |
59 | |
60 | OFPT_FLOW_REMOVED / NXT_FLOW_REMOVED | |
61 | OFPRR_IDLE_TIMEOUT yes --- | |
62 | OFPRR_HARD_TIMEOUT yes --- | |
63 | OFPRR_DELETE yes --- | |
64 | ||
65 | OFPT_PORT_STATUS | |
66 | OFPPR_ADD yes yes | |
67 | OFPPR_DELETE yes yes | |
68 | OFPPR_MODIFY yes yes | |
542cc9bb | 69 | ``` |
80d5aefd BP |
70 | |
71 | The NXT_SET_ASYNC_CONFIG message directly sets all of the values in | |
72 | this table for the current connection. The | |
73 | OFPC_INVALID_TTL_TO_CONTROLLER bit in the OFPT_SET_CONFIG message | |
74 | controls the setting for OFPR_INVALID_TTL for the "master" role. | |
75 | ||
76 | ||
77 | OFPAT_ENQUEUE | |
78 | ============= | |
82172632 EJ |
79 | |
80 | The OpenFlow 1.0 specification requires the output port of the OFPAT_ENQUEUE | |
81 | action to "refer to a valid physical port (i.e. < OFPP_MAX) or OFPP_IN_PORT". | |
82 | Although OFPP_LOCAL is not less than OFPP_MAX, it is an 'internal' port which | |
83 | can have QoS applied to it in Linux. Since we allow the OFPAT_ENQUEUE to apply | |
84 | to 'internal' ports whose port numbers are less than OFPP_MAX, we interpret | |
85 | OFPP_LOCAL as a physical port and support OFPAT_ENQUEUE on it as well. | |
86 | ||
d31f1109 | 87 | |
12442ec5 BP |
88 | OFPT_FLOW_MOD |
89 | ============= | |
90 | ||
3432cb4e BP |
91 | The OpenFlow specification for the behavior of OFPT_FLOW_MOD is |
92 | confusing. The following tables summarize the Open vSwitch | |
12442ec5 BP |
93 | implementation of its behavior in the following categories: |
94 | ||
542cc9bb TG |
95 | - "match on priority": Whether the flow_mod acts only on flows |
96 | whose priority matches that included in the flow_mod message. | |
12442ec5 | 97 | |
542cc9bb TG |
98 | - "match on out_port": Whether the flow_mod acts only on flows |
99 | that output to the out_port included in the flow_mod message (if | |
100 | out_port is not OFPP_NONE). OpenFlow 1.1 and later have a | |
101 | similar feature (not listed separately here) for out_group. | |
3432cb4e | 102 | |
542cc9bb TG |
103 | - "match on flow_cookie": Whether the flow_mod acts only on flows |
104 | whose flow_cookie matches an optional controller-specified value | |
105 | and mask. | |
12442ec5 | 106 | |
542cc9bb TG |
107 | - "updates flow_cookie": Whether the flow_mod changes the |
108 | flow_cookie of the flow or flows that it matches to the | |
109 | flow_cookie included in the flow_mod message. | |
12442ec5 | 110 | |
542cc9bb TG |
111 | - "updates OFPFF_ flags": Whether the flow_mod changes the |
112 | OFPFF_SEND_FLOW_REM flag of the flow or flows that it matches to | |
113 | the setting included in the flags of the flow_mod message. | |
12442ec5 | 114 | |
542cc9bb TG |
115 | - "honors OFPFF_CHECK_OVERLAP": Whether the OFPFF_CHECK_OVERLAP |
116 | flag in the flow_mod is significant. | |
12442ec5 | 117 | |
542cc9bb TG |
118 | - "updates idle_timeout" and "updates hard_timeout": Whether the |
119 | idle_timeout and hard_timeout in the flow_mod, respectively, | |
120 | have an effect on the flow or flows matched by the flow_mod. | |
12442ec5 | 121 | |
542cc9bb TG |
122 | - "updates idle timer": Whether the flow_mod resets the per-flow |
123 | timer that measures how long a flow has been idle. | |
12442ec5 | 124 | |
542cc9bb TG |
125 | - "updates hard timer": Whether the flow_mod resets the per-flow |
126 | timer that measures how long it has been since a flow was | |
127 | modified. | |
12442ec5 | 128 | |
542cc9bb TG |
129 | - "zeros counters": Whether the flow_mod resets per-flow packet |
130 | and byte counters to zero. | |
12442ec5 | 131 | |
542cc9bb TG |
132 | - "may add a new flow": Whether the flow_mod may add a new flow to |
133 | the flow table. (Obviously this is always true for "add" | |
134 | commands but in some OpenFlow versions "modify" and | |
135 | "modify-strict" can also add new flows.) | |
3432cb4e | 136 | |
542cc9bb TG |
137 | - "sends flow_removed message": Whether the flow_mod generates a |
138 | flow_removed message for the flow or flows that it affects. | |
12442ec5 BP |
139 | |
140 | An entry labeled "yes" means that the flow mod type does have the | |
141 | indicated behavior, "---" means that it does not, an empty cell means | |
142 | that the property is not applicable, and other values are explained | |
143 | below the table. | |
144 | ||
3432cb4e BP |
145 | OpenFlow 1.0 |
146 | ------------ | |
147 | ||
542cc9bb | 148 | ``` |
12442ec5 BP |
149 | MODIFY DELETE |
150 | ADD MODIFY STRICT DELETE STRICT | |
151 | === ====== ====== ====== ====== | |
3432cb4e | 152 | match on priority yes --- yes --- yes |
906087ee | 153 | match on out_port --- --- --- yes yes |
3432cb4e BP |
154 | match on flow_cookie --- --- --- --- --- |
155 | match on table_id --- --- --- --- --- | |
156 | controller chooses table_id --- --- --- | |
12442ec5 BP |
157 | updates flow_cookie yes yes yes |
158 | updates OFPFF_SEND_FLOW_REM yes + + | |
159 | honors OFPFF_CHECK_OVERLAP yes + + | |
160 | updates idle_timeout yes + + | |
161 | updates hard_timeout yes + + | |
162 | resets idle timer yes + + | |
163 | resets hard timer yes yes yes | |
164 | zeros counters yes + + | |
3432cb4e BP |
165 | may add a new flow yes yes yes |
166 | sends flow_removed message --- --- --- % % | |
167 | ||
168 | (+) "modify" and "modify-strict" only take these actions when they | |
169 | create a new flow, not when they update an existing flow. | |
170 | ||
171 | (%) "delete" and "delete_strict" generates a flow_removed message if | |
172 | the deleted flow or flows have the OFPFF_SEND_FLOW_REM flag set. | |
173 | (Each controller can separately control whether it wants to | |
174 | receive the generated messages.) | |
542cc9bb | 175 | ``` |
3432cb4e BP |
176 | |
177 | OpenFlow 1.1 | |
178 | ------------ | |
179 | ||
180 | OpenFlow 1.1 makes these changes: | |
181 | ||
542cc9bb TG |
182 | - The controller now must specify the table_id of the flow match |
183 | searched and into which a flow may be inserted. Behavior for a | |
184 | table_id of 255 is undefined. | |
3432cb4e | 185 | |
542cc9bb | 186 | - A flow_mod, except an "add", can now match on the flow_cookie. |
3432cb4e | 187 | |
542cc9bb TG |
188 | - When a flow_mod matches on the flow_cookie, "modify" and |
189 | "modify-strict" never insert a new flow. | |
3432cb4e | 190 | |
542cc9bb | 191 | ``` |
3432cb4e BP |
192 | MODIFY DELETE |
193 | ADD MODIFY STRICT DELETE STRICT | |
194 | === ====== ====== ====== ====== | |
195 | match on priority yes --- yes --- yes | |
196 | match on out_port --- --- --- yes yes | |
197 | match on flow_cookie --- yes yes yes yes | |
198 | match on table_id yes yes yes yes yes | |
199 | controller chooses table_id yes yes yes | |
200 | updates flow_cookie yes --- --- | |
201 | updates OFPFF_SEND_FLOW_REM yes + + | |
202 | honors OFPFF_CHECK_OVERLAP yes + + | |
203 | updates idle_timeout yes + + | |
204 | updates hard_timeout yes + + | |
205 | resets idle timer yes + + | |
206 | resets hard timer yes yes yes | |
207 | zeros counters yes + + | |
208 | may add a new flow yes # # | |
12442ec5 BP |
209 | sends flow_removed message --- --- --- % % |
210 | ||
211 | (+) "modify" and "modify-strict" only take these actions when they | |
212 | create a new flow, not when they update an existing flow. | |
213 | ||
214 | (%) "delete" and "delete_strict" generates a flow_removed message if | |
215 | the deleted flow or flows have the OFPFF_SEND_FLOW_REM flag set. | |
216 | (Each controller can separately control whether it wants to | |
217 | receive the generated messages.) | |
218 | ||
3432cb4e BP |
219 | (#) "modify" and "modify-strict" only add a new flow if the flow_mod |
220 | does not match on any bits of the flow cookie | |
542cc9bb | 221 | ``` |
3432cb4e BP |
222 | |
223 | OpenFlow 1.2 | |
224 | ------------ | |
225 | ||
226 | OpenFlow 1.2 makes these changes: | |
227 | ||
542cc9bb TG |
228 | - Only "add" commands ever add flows, "modify" and "modify-strict" |
229 | never do. | |
3432cb4e | 230 | |
542cc9bb TG |
231 | - A new flag OFPFF_RESET_COUNTS now controls whether "modify" and |
232 | "modify-strict" reset counters, whereas previously they never | |
233 | reset counters (except when they inserted a new flow). | |
3432cb4e | 234 | |
542cc9bb | 235 | ``` |
3432cb4e BP |
236 | MODIFY DELETE |
237 | ADD MODIFY STRICT DELETE STRICT | |
238 | === ====== ====== ====== ====== | |
239 | match on priority yes --- yes --- yes | |
240 | match on out_port --- --- --- yes yes | |
241 | match on flow_cookie --- yes yes yes yes | |
242 | match on table_id yes yes yes yes yes | |
243 | controller chooses table_id yes yes yes | |
244 | updates flow_cookie yes --- --- | |
245 | updates OFPFF_SEND_FLOW_REM yes --- --- | |
246 | honors OFPFF_CHECK_OVERLAP yes --- --- | |
247 | updates idle_timeout yes --- --- | |
248 | updates hard_timeout yes --- --- | |
249 | resets idle timer yes --- --- | |
250 | resets hard timer yes yes yes | |
251 | zeros counters yes & & | |
252 | may add a new flow yes --- --- | |
253 | sends flow_removed message --- --- --- % % | |
254 | ||
255 | (%) "delete" and "delete_strict" generates a flow_removed message if | |
256 | the deleted flow or flows have the OFPFF_SEND_FLOW_REM flag set. | |
257 | (Each controller can separately control whether it wants to | |
258 | receive the generated messages.) | |
259 | ||
260 | (&) "modify" and "modify-strict" reset counters if the | |
261 | OFPFF_RESET_COUNTS flag is specified. | |
542cc9bb | 262 | ``` |
3432cb4e BP |
263 | |
264 | OpenFlow 1.3 | |
265 | ------------ | |
266 | ||
267 | OpenFlow 1.3 makes these changes: | |
268 | ||
542cc9bb TG |
269 | - Behavior for a table_id of 255 is now defined, for "delete" and |
270 | "delete-strict" commands, as meaning to delete from all tables. | |
271 | A table_id of 255 is now explicitly invalid for other commands. | |
3432cb4e | 272 | |
542cc9bb TG |
273 | - New flags OFPFF_NO_PKT_COUNTS and OFPFF_NO_BYT_COUNTS for "add" |
274 | operations. | |
3432cb4e BP |
275 | |
276 | The table for 1.3 is the same as the one shown above for 1.2. | |
277 | ||
12442ec5 | 278 | |
c37c0382 AC |
279 | OpenFlow 1.4 |
280 | ------------ | |
281 | ||
ca26eb44 RB |
282 | OpenFlow 1.4 adds the "importance" field to flow_mods, but it does not |
283 | explicitly specify which kinds of flow_mods set the importance.For | |
284 | consistency, Open vSwitch uses the same rule for importance as for | |
285 | idle_timeout and hard_timeout, that is, only an "ADD" flow_mod sets | |
286 | the importance. (This issue has been filed with the ONF as EXT-496.) | |
c37c0382 | 287 | |
4d197ebb BP |
288 | OFPT_PACKET_IN |
289 | ============== | |
290 | ||
291 | The OpenFlow 1.1 specification for OFPT_PACKET_IN is confusing. The | |
292 | definition in OF1.1 openflow.h is[*]: | |
293 | ||
542cc9bb | 294 | ``` |
4d197ebb BP |
295 | /* Packet received on port (datapath -> controller). */ |
296 | struct ofp_packet_in { | |
297 | struct ofp_header header; | |
298 | uint32_t buffer_id; /* ID assigned by datapath. */ | |
299 | uint32_t in_port; /* Port on which frame was received. */ | |
300 | uint32_t in_phy_port; /* Physical Port on which frame was received. */ | |
301 | uint16_t total_len; /* Full length of frame. */ | |
302 | uint8_t reason; /* Reason packet is being sent (one of OFPR_*) */ | |
303 | uint8_t table_id; /* ID of the table that was looked up */ | |
304 | uint8_t data[0]; /* Ethernet frame, halfway through 32-bit word, | |
305 | so the IP header is 32-bit aligned. The | |
306 | amount of data is inferred from the length | |
307 | field in the header. Because of padding, | |
308 | offsetof(struct ofp_packet_in, data) == | |
309 | sizeof(struct ofp_packet_in) - 2. */ | |
310 | }; | |
311 | OFP_ASSERT(sizeof(struct ofp_packet_in) == 24); | |
542cc9bb | 312 | ``` |
4d197ebb BP |
313 | |
314 | The confusing part is the comment on the data[] member. This comment | |
315 | is a leftover from OF1.0 openflow.h, in which the comment was correct: | |
316 | sizeof(struct ofp_packet_in) is 20 in OF1.0 and offsetof(struct | |
317 | ofp_packet_in, data) is 18. When OF1.1 was written, the structure | |
318 | members were changed but the comment was carelessly not updated, and | |
319 | the comment became wrong: sizeof(struct ofp_packet_in) and | |
320 | offsetof(struct ofp_packet_in, data) are both 24 in OF1.1. | |
321 | ||
322 | That leaves the question of how to implement ofp_packet_in in OF1.1. | |
323 | The OpenFlow reference implementation for OF1.1 does not include any | |
324 | padding, that is, the first byte of the encapsulated frame immediately | |
325 | follows the 'table_id' member without a gap. Open vSwitch therefore | |
326 | implements it the same way for compatibility. | |
327 | ||
328 | For an earlier discussion, please see the thread archived at: | |
329 | https://mailman.stanford.edu/pipermail/openflow-discuss/2011-August/002604.html | |
330 | ||
331 | [*] The quoted definition is directly from OF1.1. Definitions used | |
332 | inside OVS omit the 8-byte ofp_header members, so the sizes in | |
333 | this discussion are 8 bytes larger than those declared in OVS | |
334 | header files. | |
335 | ||
336 | ||
df778240 BP |
337 | VLAN Matching |
338 | ============= | |
339 | ||
340 | The 802.1Q VLAN header causes more trouble than any other 4 bytes in | |
341 | networking. More specifically, three versions of OpenFlow and Open | |
342 | vSwitch have among them four different ways to match the contents and | |
343 | presence of the VLAN header. The following table describes how each | |
344 | version works. | |
345 | ||
346 | Match NXM OF1.0 OF1.1 OF1.2 | |
347 | ----- --------- ----------- ----------- ------------ | |
348 | [1] 0000/0000 ????/1,??/? ????/1,??/? 0000/0000,-- | |
349 | [2] 0000/ffff ffff/0,??/? ffff/0,??/? 0000/ffff,-- | |
350 | [3] 1xxx/1fff 0xxx/0,??/1 0xxx/0,??/1 1xxx/ffff,-- | |
351 | [4] z000/f000 ????/1,0y/0 fffe/0,0y/0 1000/1000,0y | |
352 | [5] zxxx/ffff 0xxx/0,0y/0 0xxx/0,0y/0 1xxx/ffff,0y | |
353 | [6] 0000/0fff <none> <none> <none> | |
354 | [7] 0000/f000 <none> <none> <none> | |
355 | [8] 0000/efff <none> <none> <none> | |
356 | [9] 1001/1001 <none> <none> 1001/1001,-- | |
357 | [10] 3000/3000 <none> <none> <none> | |
358 | ||
359 | Each column is interpreted as follows. | |
360 | ||
542cc9bb | 361 | - Match: See the list below. |
df778240 | 362 | |
542cc9bb TG |
363 | - NXM: xxxx/yyyy means NXM_OF_VLAN_TCI_W with value xxxx and mask |
364 | yyyy. A mask of 0000 is equivalent to omitting | |
365 | NXM_OF_VLAN_TCI(_W), a mask of ffff is equivalent to | |
366 | NXM_OF_VLAN_TCI. | |
df778240 | 367 | |
542cc9bb TG |
368 | - OF1.0 and OF1.1: wwww/x,yy/z means dl_vlan wwww, OFPFW_DL_VLAN |
369 | x, dl_vlan_pcp yy, and OFPFW_DL_VLAN_PCP z. ? means that the | |
370 | given nibble is ignored (and conventionally 0 for wwww or yy, | |
371 | conventionally 1 for x or z). <none> means that the given match | |
372 | is not supported. | |
df778240 | 373 | |
542cc9bb TG |
374 | - OF1.2: xxxx/yyyy,zz means OXM_OF_VLAN_VID_W with value xxxx and |
375 | mask yyyy, and OXM_OF_VLAN_PCP (which is not maskable) with | |
376 | value zz. A mask of 0000 is equivalent to omitting | |
377 | OXM_OF_VLAN_VID(_W), a mask of ffff is equivalent to | |
378 | OXM_OF_VLAN_VID. -- means that OXM_OF_VLAN_PCP is omitted. | |
379 | <none> means that the given match is not supported. | |
df778240 BP |
380 | |
381 | The matches are: | |
382 | ||
383 | [1] Matches any packet, that is, one without an 802.1Q header or with | |
384 | an 802.1Q header with any TCI value. | |
385 | ||
386 | [2] Matches only packets without an 802.1Q header. | |
387 | ||
388 | NXM: Any match with (vlan_tci == 0) and (vlan_tci_mask & 0x1000) | |
389 | != 0 is equivalent to the one listed in the table. | |
390 | ||
391 | OF1.0: The spec doesn't define behavior if dl_vlan is set to | |
392 | 0xffff and OFPFW_DL_VLAN_PCP is not set. | |
393 | ||
394 | OF1.1: The spec says explicitly to ignore dl_vlan_pcp when | |
395 | dl_vlan is set to 0xffff. | |
396 | ||
397 | OF1.2: The spec doesn't say what should happen if (vlan_vid == 0) | |
398 | and (vlan_vid_mask & 0x1000) != 0 but (vlan_vid_mask != 0x1000), | |
399 | but it would be straightforward to also interpret as [2]. | |
400 | ||
401 | [3] Matches only packets that have an 802.1Q header with VID xxx (and | |
402 | any PCP). | |
403 | ||
404 | [4] Matches only packets that have an 802.1Q header with PCP y (and | |
405 | any VID). | |
406 | ||
407 | NXM: z is ((y << 1) | 1). | |
408 | ||
409 | OF1.0: The spec isn't very clear, but OVS implements it this way. | |
410 | ||
411 | OF1.2: Presumably other masks such that (vlan_vid_mask & 0x1fff) | |
412 | == 0x1000 would also work, but the spec doesn't define their | |
413 | behavior. | |
414 | ||
415 | [5] Matches only packets that have an 802.1Q header with VID xxx and | |
416 | PCP y. | |
417 | ||
418 | NXM: z is ((y << 1) | 1). | |
419 | ||
420 | OF1.2: Presumably other masks such that (vlan_vid_mask & 0x1fff) | |
421 | == 0x1fff would also work. | |
422 | ||
423 | [6] Matches packets with no 802.1Q header or with an 802.1Q header | |
424 | with a VID of 0. Only possible with NXM. | |
425 | ||
426 | [7] Matches packets with no 802.1Q header or with an 802.1Q header | |
427 | with a PCP of 0. Only possible with NXM. | |
428 | ||
429 | [8] Matches packets with no 802.1Q header or with an 802.1Q header | |
430 | with both VID and PCP of 0. Only possible with NXM. | |
431 | ||
432 | [9] Matches only packets that have an 802.1Q header with an | |
433 | odd-numbered VID (and any PCP). Only possible with NXM and | |
434 | OF1.2. (This is just an example; one can match on any desired | |
435 | VID bit pattern.) | |
436 | ||
437 | [10] Matches only packets that have an 802.1Q header with an | |
438 | odd-numbered PCP (and any VID). Only possible with NXM. (This | |
439 | is just an example; one can match on any desired VID bit | |
440 | pattern.) | |
441 | ||
442 | Additional notes: | |
443 | ||
542cc9bb TG |
444 | - OF1.2: The top three bits of OXM_OF_VLAN_VID are fixed to zero, |
445 | so bits 13, 14, and 15 in the masks listed in the table may be | |
446 | set to arbitrary values, as long as the corresponding value bits | |
447 | are also zero. The suggested ffff mask for [2], [3], and [5] | |
448 | allows a shorter OXM representation (the mask is omitted) than | |
449 | the minimal 1fff mask. | |
df778240 BP |
450 | |
451 | ||
f66b87de BP |
452 | Flow Cookies |
453 | ============ | |
454 | ||
455 | OpenFlow 1.0 and later versions have the concept of a "flow cookie", | |
456 | which is a 64-bit integer value attached to each flow. The treatment | |
457 | of the flow cookie has varied greatly across OpenFlow versions, | |
458 | however. | |
459 | ||
460 | In OpenFlow 1.0: | |
461 | ||
542cc9bb | 462 | - OFPFC_ADD set the cookie in the flow that it added. |
f66b87de | 463 | |
542cc9bb TG |
464 | - OFPFC_MODIFY and OFPFC_MODIFY_STRICT updated the cookie for |
465 | the flow or flows that it modified. | |
f66b87de | 466 | |
542cc9bb | 467 | - OFPST_FLOW messages included the flow cookie. |
f66b87de | 468 | |
542cc9bb TG |
469 | - OFPT_FLOW_REMOVED messages reported the cookie of the flow |
470 | that was removed. | |
f66b87de BP |
471 | |
472 | OpenFlow 1.1 made the following changes: | |
473 | ||
542cc9bb TG |
474 | - Flow mod operations OFPFC_MODIFY, OFPFC_MODIFY_STRICT, |
475 | OFPFC_DELETE, and OFPFC_DELETE_STRICT, plus flow stats | |
476 | requests and aggregate stats requests, gained the ability to | |
477 | match on flow cookies with an arbitrary mask. | |
f66b87de | 478 | |
542cc9bb TG |
479 | - OFPFC_MODIFY and OFPFC_MODIFY_STRICT were changed to add a |
480 | new flow, in the case of no match, only if the flow table | |
481 | modification operation did not match on the cookie field. | |
482 | (In OpenFlow 1.0, modify operations always added a new flow | |
483 | when there was no match.) | |
f66b87de | 484 | |
542cc9bb TG |
485 | - OFPFC_MODIFY and OFPFC_MODIFY_STRICT no longer updated flow |
486 | cookies. | |
f66b87de BP |
487 | |
488 | OpenFlow 1.2 made the following changes: | |
489 | ||
542cc9bb TG |
490 | - OFPC_MODIFY and OFPFC_MODIFY_STRICT were changed to never |
491 | add a new flow, regardless of whether the flow cookie was | |
492 | used for matching. | |
f66b87de BP |
493 | |
494 | Open vSwitch support for OpenFlow 1.0 implements the OpenFlow 1.0 | |
495 | behavior with the following extensions: | |
496 | ||
542cc9bb TG |
497 | - An NXM extension field NXM_NX_COOKIE(_W) allows the NXM |
498 | versions of OFPFC_MODIFY, OFPFC_MODIFY_STRICT, OFPFC_DELETE, | |
499 | and OFPFC_DELETE_STRICT flow_mods, plus flow stats requests | |
500 | and aggregate stats requests, to match on flow cookies with | |
501 | arbitrary masks. This is much like the equivalent OpenFlow | |
502 | 1.1 feature. | |
503 | ||
504 | - Like OpenFlow 1.1, OFPC_MODIFY and OFPFC_MODIFY_STRICT add a | |
505 | new flow if there is no match and the mask is zero (or not | |
506 | given). | |
507 | ||
508 | - The "cookie" field in OFPT_FLOW_MOD and NXT_FLOW_MOD messages | |
509 | is used as the cookie value for OFPFC_ADD commands, as | |
510 | described in OpenFlow 1.0. For OFPFC_MODIFY and | |
511 | OFPFC_MODIFY_STRICT commands, the "cookie" field is used as a | |
512 | new cookie for flows that match unless it is UINT64_MAX, in | |
513 | which case the flow's cookie is not updated. | |
514 | ||
515 | - NXT_PACKET_IN (the Nicira extended version of | |
516 | OFPT_PACKET_IN) reports the cookie of the rule that | |
517 | generated the packet, or all-1-bits if no rule generated the | |
518 | packet. (Older versions of OVS used all-0-bits instead of | |
519 | all-1-bits.) | |
f66b87de | 520 | |
623e1caf JP |
521 | The following table shows the handling of different protocols when |
522 | receiving OFPFC_MODIFY and OFPFC_MODIFY_STRICT messages. A mask of 0 | |
523 | indicates either an explicit mask of zero or an implicit one by not | |
524 | specifying the NXM_NX_COOKIE(_W) field. | |
525 | ||
542cc9bb | 526 | ``` |
623e1caf JP |
527 | Match Update Add on miss Add on miss |
528 | cookie cookie mask!=0 mask==0 | |
529 | ====== ====== =========== =========== | |
530 | OpenFlow 1.0 no yes <always add on miss> | |
531 | OpenFlow 1.1 yes no no yes | |
532 | OpenFlow 1.2 yes no no no | |
533 | NXM yes yes* no yes | |
534 | ||
535 | * Updates the flow's cookie unless the "cookie" field is UINT64_MAX. | |
542cc9bb | 536 | ``` |
f66b87de | 537 | |
66abb12b BP |
538 | Multiple Table Support |
539 | ====================== | |
540 | ||
541 | OpenFlow 1.0 has only rudimentary support for multiple flow tables. | |
542 | Notably, OpenFlow 1.0 does not allow the controller to specify the | |
543 | flow table to which a flow is to be added. Open vSwitch adds an | |
544 | extension for this purpose, which is enabled on a per-OpenFlow | |
545 | connection basis using the NXT_FLOW_MOD_TABLE_ID message. When the | |
546 | extension is enabled, the upper 8 bits of the 'command' member in an | |
547 | OFPT_FLOW_MOD or NXT_FLOW_MOD message designates the table to which a | |
548 | flow is to be added. | |
549 | ||
550 | The Open vSwitch software switch implementation offers 255 flow | |
551 | tables. On packet ingress, only the first flow table (table 0) is | |
552 | searched, and the contents of the remaining tables are not considered | |
553 | in any way. Tables other than table 0 only come into play when an | |
554 | NXAST_RESUBMIT_TABLE action specifies another table to search. | |
555 | ||
556 | Tables 128 and above are reserved for use by the switch itself. | |
557 | Controllers should use only tables 0 through 127. | |
558 | ||
559 | ||
d31f1109 JP |
560 | IPv6 |
561 | ==== | |
562 | ||
563 | Open vSwitch supports stateless handling of IPv6 packets. Flows can be | |
564 | written to support matching TCP, UDP, and ICMPv6 headers within an IPv6 | |
685a51a5 JP |
565 | packet. Deeper matching of some Neighbor Discovery messages is also |
566 | supported. | |
d31f1109 JP |
567 | |
568 | IPv6 was not designed to interact well with middle-boxes. This, | |
569 | combined with Open vSwitch's stateless nature, have affected the | |
570 | processing of IPv6 traffic, which is detailed below. | |
571 | ||
572 | Extension Headers | |
573 | ----------------- | |
574 | ||
575 | The base IPv6 header is incredibly simple with the intention of only | |
576 | containing information relevant for routing packets between two | |
577 | endpoints. IPv6 relies heavily on the use of extension headers to | |
578 | provide any other functionality. Unfortunately, the extension headers | |
579 | were designed in such a way that it is impossible to move to the next | |
580 | header (including the layer-4 payload) unless the current header is | |
581 | understood. | |
582 | ||
583 | Open vSwitch will process the following extension headers and continue | |
584 | to the next header: | |
585 | ||
542cc9bb TG |
586 | * Fragment (see the next section) |
587 | * AH (Authentication Header) | |
588 | * Hop-by-Hop Options | |
589 | * Routing | |
590 | * Destination Options | |
d31f1109 JP |
591 | |
592 | When a header is encountered that is not in that list, it is considered | |
593 | "terminal". A terminal header's IPv6 protocol value is stored in | |
594 | "nw_proto" for matching purposes. If a terminal header is TCP, UDP, or | |
595 | ICMPv6, the packet will be further processed in an attempt to extract | |
596 | layer-4 information. | |
597 | ||
598 | Fragments | |
599 | --------- | |
600 | ||
601 | IPv6 requires that every link in the internet have an MTU of 1280 octets | |
602 | or greater (RFC 2460). As such, a terminal header (as described above in | |
603 | "Extension Headers") in the first fragment should generally be | |
604 | reachable. In this case, the terminal header's IPv6 protocol type is | |
605 | stored in the "nw_proto" field for matching purposes. If a terminal | |
606 | header cannot be found in the first fragment (one with a fragment offset | |
607 | of zero), the "nw_proto" field is set to 0. Subsequent fragments (those | |
608 | with a non-zero fragment offset) have the "nw_proto" field set to the | |
609 | IPv6 protocol type for fragments (44). | |
610 | ||
611 | Jumbograms | |
612 | ---------- | |
613 | ||
614 | An IPv6 jumbogram (RFC 2675) is a packet containing a payload longer | |
615 | than 65,535 octets. A jumbogram is only relevant in subnets with a link | |
616 | MTU greater than 65,575 octets, and are not required to be supported on | |
617 | nodes that do not connect to link with such large MTUs. Currently, Open | |
618 | vSwitch doesn't process jumbograms. | |
619 | ||
620 | ||
946350dc BP |
621 | In-Band Control |
622 | =============== | |
623 | ||
56e9c3b9 BP |
624 | Motivation |
625 | ---------- | |
626 | ||
627 | An OpenFlow switch must establish and maintain a TCP network | |
628 | connection to its controller. There are two basic ways to categorize | |
629 | the network that this connection traverses: either it is completely | |
630 | separate from the one that the switch is otherwise controlling, or its | |
631 | path may overlap the network that the switch controls. We call the | |
632 | former case "out-of-band control", the latter case "in-band control". | |
633 | ||
634 | Out-of-band control has the following benefits: | |
635 | ||
542cc9bb TG |
636 | - Simplicity: Out-of-band control slightly simplifies the switch |
637 | implementation. | |
56e9c3b9 | 638 | |
542cc9bb TG |
639 | - Reliability: Excessive switch traffic volume cannot interfere |
640 | with control traffic. | |
56e9c3b9 | 641 | |
542cc9bb TG |
642 | - Integrity: Machines not on the control network cannot |
643 | impersonate a switch or a controller. | |
56e9c3b9 | 644 | |
542cc9bb TG |
645 | - Confidentiality: Machines not on the control network cannot |
646 | snoop on control traffic. | |
56e9c3b9 BP |
647 | |
648 | In-band control, on the other hand, has the following advantages: | |
649 | ||
542cc9bb TG |
650 | - No dedicated port: There is no need to dedicate a physical |
651 | switch port to control, which is important on switches that have | |
652 | few ports (e.g. wireless routers, low-end embedded platforms). | |
56e9c3b9 | 653 | |
542cc9bb TG |
654 | - No dedicated network: There is no need to build and maintain a |
655 | separate control network. This is important in many | |
656 | environments because it reduces proliferation of switches and | |
657 | wiring. | |
56e9c3b9 BP |
658 | |
659 | Open vSwitch supports both out-of-band and in-band control. This | |
660 | section describes the principles behind in-band control. See the | |
661 | description of the Controller table in ovs-vswitchd.conf.db(5) to | |
662 | configure OVS for in-band control. | |
663 | ||
664 | Principles | |
665 | ---------- | |
666 | ||
667 | The fundamental principle of in-band control is that an OpenFlow | |
668 | switch must recognize and switch control traffic without involving the | |
669 | OpenFlow controller. All the details of implementing in-band control | |
670 | are special cases of this principle. | |
671 | ||
672 | The rationale for this principle is simple. If the switch does not | |
673 | handle in-band control traffic itself, then it will be caught in a | |
674 | contradiction: it must contact the controller, but it cannot, because | |
675 | only the controller can set up the flows that are needed to contact | |
676 | the controller. | |
677 | ||
678 | The following points describe important special cases of this | |
679 | principle. | |
680 | ||
542cc9bb TG |
681 | - In-band control must be implemented regardless of whether the |
682 | switch is connected. | |
683 | ||
684 | It is tempting to implement the in-band control rules only when | |
685 | the switch is not connected to the controller, using the | |
686 | reasoning that the controller should have complete control once | |
687 | it has established a connection with the switch. | |
688 | ||
689 | This does not work in practice. Consider the case where the | |
690 | switch is connected to the controller. Occasionally it can | |
691 | happen that the controller forgets or otherwise needs to obtain | |
692 | the MAC address of the switch. To do so, the controller sends a | |
693 | broadcast ARP request. A switch that implements the in-band | |
694 | control rules only when it is disconnected will then send an | |
695 | OFPT_PACKET_IN message up to the controller. The controller will | |
696 | be unable to respond, because it does not know the MAC address of | |
697 | the switch. This is a deadlock situation that can only be | |
698 | resolved by the switch noticing that its connection to the | |
699 | controller has hung and reconnecting. | |
700 | ||
701 | - In-band control must override flows set up by the controller. | |
702 | ||
703 | It is reasonable to assume that flows set up by the OpenFlow | |
704 | controller should take precedence over in-band control, on the | |
705 | basis that the controller should be in charge of the switch. | |
706 | ||
707 | Again, this does not work in practice. Reasonable controller | |
708 | implementations may set up a "last resort" fallback rule that | |
709 | wildcards every field and, e.g., sends it up to the controller or | |
710 | discards it. If a controller does that, then it will isolate | |
711 | itself from the switch. | |
712 | ||
713 | - The switch must recognize all control traffic. | |
714 | ||
715 | The fundamental principle of in-band control states, in part, | |
716 | that a switch must recognize control traffic without involving | |
717 | the OpenFlow controller. More specifically, the switch must | |
718 | recognize *all* control traffic. "False negatives", that is, | |
719 | packets that constitute control traffic but that the switch does | |
720 | not recognize as control traffic, lead to control traffic storms. | |
721 | ||
722 | Consider an OpenFlow switch that only recognizes control packets | |
723 | sent to or from that switch. Now suppose that two switches of | |
724 | this type, named A and B, are connected to ports on an Ethernet | |
725 | hub (not a switch) and that an OpenFlow controller is connected | |
726 | to a third hub port. In this setup, control traffic sent by | |
727 | switch A will be seen by switch B, which will send it to the | |
728 | controller as part of an OFPT_PACKET_IN message. Switch A will | |
729 | then see the OFPT_PACKET_IN message's packet, re-encapsulate it | |
730 | in another OFPT_PACKET_IN, and send it to the controller. Switch | |
731 | B will then see that OFPT_PACKET_IN, and so on in an infinite | |
732 | loop. | |
733 | ||
734 | Incidentally, the consequences of "false positives", where | |
735 | packets that are not control traffic are nevertheless recognized | |
736 | as control traffic, are much less severe. The controller will | |
737 | not be able to control their behavior, but the network will | |
738 | remain in working order. False positives do constitute a | |
739 | security problem. | |
740 | ||
741 | - The switch should use echo-requests to detect disconnection. | |
742 | ||
743 | TCP will notice that a connection has hung, but this can take a | |
744 | considerable amount of time. For example, with default settings | |
745 | the Linux kernel TCP implementation will retransmit for between | |
746 | 13 and 30 minutes, depending on the connection's retransmission | |
747 | timeout, according to kernel documentation. This is far too long | |
748 | for a switch to be disconnected, so an OpenFlow switch should | |
749 | implement its own connection timeout. OpenFlow OFPT_ECHO_REQUEST | |
750 | messages are the best way to do this, since they test the | |
751 | OpenFlow connection itself. | |
56e9c3b9 BP |
752 | |
753 | Implementation | |
754 | -------------- | |
755 | ||
756 | This section describes how Open vSwitch implements in-band control. | |
757 | Correctly implementing in-band control has proven difficult due to its | |
758 | many subtleties, and has thus gone through many iterations. Please | |
759 | read through and understand the reasoning behind the chosen rules | |
760 | before making modifications. | |
761 | ||
762 | Open vSwitch implements in-band control as "hidden" flows, that is, | |
763 | flows that are not visible through OpenFlow, and at a higher priority | |
764 | than wildcarded flows can be set up through OpenFlow. This is done so | |
765 | that the OpenFlow controller cannot interfere with them and possibly | |
766 | break connectivity with its switches. It is possible to see all | |
767 | flows, including in-band ones, with the ovs-appctl "bridge/dump-flows" | |
768 | command. | |
946350dc BP |
769 | |
770 | The Open vSwitch implementation of in-band control can hide traffic to | |
771 | arbitrary "remotes", where each remote is one TCP port on one IP address. | |
772 | Currently the remotes are automatically configured as the in-band OpenFlow | |
773 | controllers plus the OVSDB managers, if any. (The latter is a requirement | |
774 | because OVSDB managers are responsible for configuring OpenFlow controllers, | |
775 | so if the manager cannot be reached then OpenFlow cannot be reconfigured.) | |
776 | ||
777 | The following rules (with the OFPP_NORMAL action) are set up on any bridge | |
778 | that has any remotes: | |
779 | ||
780 | (a) DHCP requests sent from the local port. | |
781 | (b) ARP replies to the local port's MAC address. | |
782 | (c) ARP requests from the local port's MAC address. | |
783 | ||
784 | In-band also sets up the following rules for each unique next-hop MAC | |
785 | address for the remotes' IPs (the "next hop" is either the remote | |
786 | itself, if it is on a local subnet, or the gateway to reach the remote): | |
787 | ||
788 | (d) ARP replies to the next hop's MAC address. | |
789 | (e) ARP requests from the next hop's MAC address. | |
790 | ||
791 | In-band also sets up the following rules for each unique remote IP address: | |
792 | ||
793 | (f) ARP replies containing the remote's IP address as a target. | |
794 | (g) ARP requests containing the remote's IP address as a source. | |
795 | ||
796 | In-band also sets up the following rules for each unique remote (IP,port) | |
797 | pair: | |
798 | ||
799 | (h) TCP traffic to the remote's IP and port. | |
800 | (i) TCP traffic from the remote's IP and port. | |
801 | ||
802 | The goal of these rules is to be as narrow as possible to allow a | |
803 | switch to join a network and be able to communicate with the | |
804 | remotes. As mentioned earlier, these rules have higher priority | |
805 | than the controller's rules, so if they are too broad, they may | |
806 | prevent the controller from implementing its policy. As such, | |
807 | in-band actively monitors some aspects of flow and packet processing | |
808 | so that the rules can be made more precise. | |
809 | ||
810 | In-band control monitors attempts to add flows into the datapath that | |
811 | could interfere with its duties. The datapath only allows exact | |
812 | match entries, so in-band control is able to be very precise about | |
813 | the flows it prevents. Flows that miss in the datapath are sent to | |
814 | userspace to be processed, so preventing these flows from being | |
815 | cached in the "fast path" does not affect correctness. The only type | |
816 | of flow that is currently prevented is one that would prevent DHCP | |
817 | replies from being seen by the local port. For example, a rule that | |
818 | forwarded all DHCP traffic to the controller would not be allowed, | |
819 | but one that forwarded to all ports (including the local port) would. | |
820 | ||
821 | As mentioned earlier, packets that miss in the datapath are sent to | |
822 | the userspace for processing. The userspace has its own flow table, | |
823 | the "classifier", so in-band checks whether any special processing | |
824 | is needed before the classifier is consulted. If a packet is a DHCP | |
825 | response to a request from the local port, the packet is forwarded to | |
826 | the local port, regardless of the flow table. Note that this requires | |
827 | L7 processing of DHCP replies to determine whether the 'chaddr' field | |
828 | matches the MAC address of the local port. | |
829 | ||
830 | It is interesting to note that for an L3-based in-band control | |
831 | mechanism, the majority of rules are devoted to ARP traffic. At first | |
832 | glance, some of these rules appear redundant. However, each serves an | |
833 | important role. First, in order to determine the MAC address of the | |
834 | remote side (controller or gateway) for other ARP rules, we must allow | |
835 | ARP traffic for our local port with rules (b) and (c). If we are | |
836 | between a switch and its connection to the remote, we have to | |
837 | allow the other switch's ARP traffic to through. This is done with | |
838 | rules (d) and (e), since we do not know the addresses of the other | |
839 | switches a priori, but do know the remote's or gateway's. Finally, | |
840 | if the remote is running in a local guest VM that is not reached | |
841 | through the local port, the switch that is connected to the VM must | |
842 | allow ARP traffic based on the remote's IP address, since it will | |
843 | not know the MAC address of the local port that is sending the traffic | |
844 | or the MAC address of the remote in the guest VM. | |
845 | ||
846 | With a few notable exceptions below, in-band should work in most | |
847 | network setups. The following are considered "supported' in the | |
848 | current implementation: | |
849 | ||
542cc9bb TG |
850 | - Locally Connected. The switch and remote are on the same |
851 | subnet. This uses rules (a), (b), (c), (h), and (i). | |
852 | ||
853 | - Reached through Gateway. The switch and remote are on | |
854 | different subnets and must go through a gateway. This uses | |
855 | rules (a), (b), (c), (h), and (i). | |
856 | ||
857 | - Between Switch and Remote. This switch is between another | |
858 | switch and the remote, and we want to allow the other | |
859 | switch's traffic through. This uses rules (d), (e), (h), and | |
860 | (i). It uses (b) and (c) indirectly in order to know the MAC | |
861 | address for rules (d) and (e). Note that DHCP for the other | |
862 | switch will not work unless an OpenFlow controller explicitly lets this | |
863 | switch pass the traffic. | |
864 | ||
865 | - Between Switch and Gateway. This switch is between another | |
866 | switch and the gateway, and we want to allow the other switch's | |
867 | traffic through. This uses the same rules and logic as the | |
868 | "Between Switch and Remote" configuration described earlier. | |
869 | ||
870 | - Remote on Local VM. The remote is a guest VM on the | |
871 | system running in-band control. This uses rules (a), (b), (c), | |
872 | (h), and (i). | |
873 | ||
874 | - Remote on Local VM with Different Networks. The remote | |
875 | is a guest VM on the system running in-band control, but the | |
876 | local port is not used to connect to the remote. For | |
877 | example, an IP address is configured on eth0 of the switch. The | |
878 | remote's VM is connected through eth1 of the switch, but an | |
879 | IP address has not been configured for that port on the switch. | |
880 | As such, the switch will use eth0 to connect to the remote, | |
881 | and eth1's rules about the local port will not work. In the | |
882 | example, the switch attached to eth0 would use rules (a), (b), | |
883 | (c), (h), and (i) on eth0. The switch attached to eth1 would use | |
884 | rules (f), (g), (h), and (i). | |
946350dc BP |
885 | |
886 | The following are explicitly *not* supported by in-band control: | |
887 | ||
542cc9bb TG |
888 | - Specify Remote by Name. Currently, the remote must be |
889 | identified by IP address. A naive approach would be to permit | |
890 | all DNS traffic. Unfortunately, this would prevent the | |
891 | controller from defining any policy over DNS. Since switches | |
892 | that are located behind us need to connect to the remote, | |
893 | in-band cannot simply add a rule that allows DNS traffic from | |
894 | the local port. The "correct" way to support this is to parse | |
895 | DNS requests to allow all traffic related to a request for the | |
896 | remote's name through. Due to the potential security | |
897 | problems and amount of processing, we decided to hold off for | |
898 | the time-being. | |
899 | ||
900 | - Differing Remotes for Switches. All switches must know | |
901 | the L3 addresses for all the remotes that other switches | |
902 | may use, since rules need to be set up to allow traffic related | |
903 | to those remotes through. See rules (f), (g), (h), and (i). | |
904 | ||
905 | - Differing Routes for Switches. In order for the switch to | |
906 | allow other switches to connect to a remote through a | |
907 | gateway, it allows the gateway's traffic through with rules (d) | |
908 | and (e). If the routes to the remote differ for the two | |
909 | switches, we will not know the MAC address of the alternate | |
910 | gateway. | |
946350dc BP |
911 | |
912 | ||
f25d0cf3 BP |
913 | Action Reproduction |
914 | =================== | |
915 | ||
916 | It seems likely that many controllers, at least at startup, use the | |
917 | OpenFlow "flow statistics" request to obtain existing flows, then | |
918 | compare the flows' actions against the actions that they expect to | |
919 | find. Before version 1.8.0, Open vSwitch always returned exact, | |
920 | byte-for-byte copies of the actions that had been added to the flow | |
921 | table. The current version of Open vSwitch does not always do this in | |
922 | some exceptional cases. This section lists the exceptions that | |
923 | controller authors must keep in mind if they compare actual actions | |
924 | against desired actions in a bytewise fashion: | |
925 | ||
542cc9bb TG |
926 | - Open vSwitch zeros padding bytes in action structures, |
927 | regardless of their values when the flows were added. | |
f25d0cf3 | 928 | |
542cc9bb TG |
929 | - Open vSwitch "normalizes" the instructions in OpenFlow 1.1 |
930 | (and later) in the following way: | |
d01c980f | 931 | |
542cc9bb TG |
932 | * OVS sorts the instructions into the following order: |
933 | Apply-Actions, Clear-Actions, Write-Actions, | |
934 | Write-Metadata, Goto-Table. | |
d01c980f | 935 | |
542cc9bb TG |
936 | * OVS drops Apply-Actions instructions that have empty |
937 | action lists. | |
d01c980f | 938 | |
542cc9bb TG |
939 | * OVS drops Write-Actions instructions that have empty |
940 | action sets. | |
d01c980f | 941 | |
f25d0cf3 BP |
942 | Please report other discrepancies, if you notice any, so that we can |
943 | fix or document them. | |
944 | ||
945 | ||
d31f1109 JP |
946 | Suggestions |
947 | =========== | |
948 | ||
949 | Suggestions to improve Open vSwitch are welcome at discuss@openvswitch.org. |