]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | .. BSD LICENSE |
2 | Copyright(c) 2016 Intel Corporation. All rights reserved. | |
3 | All rights reserved. | |
4 | ||
5 | Redistribution and use in source and binary forms, with or without | |
6 | modification, are permitted provided that the following conditions | |
7 | are met: | |
8 | ||
9 | * Redistributions of source code must retain the above copyright | |
10 | notice, this list of conditions and the following disclaimer. | |
11 | * Redistributions in binary form must reproduce the above copyright | |
12 | notice, this list of conditions and the following disclaimer in | |
13 | the documentation and/or other materials provided with the | |
14 | distribution. | |
15 | * Neither the name of Intel Corporation nor the names of its | |
16 | contributors may be used to endorse or promote products derived | |
17 | from this software without specific prior written permission. | |
18 | ||
19 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | |
20 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | |
21 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | |
22 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | |
23 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | |
24 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | |
25 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | |
26 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | |
27 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | |
28 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | |
29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
30 | ||
31 | ||
32 | L2 Forwarding Sample Application with Cache Allocation Technology (CAT) | |
33 | ======================================================================= | |
34 | ||
35 | Basic Forwarding sample application is a simple *skeleton* example of | |
36 | a forwarding application. It has been extended to make use of CAT via extended | |
37 | command line options and linking against the libpqos library. | |
38 | ||
39 | It is intended as a demonstration of the basic components of a DPDK forwarding | |
40 | application and use of the libpqos library to program CAT. | |
41 | For more detailed implementations see the L2 and L3 forwarding | |
42 | sample applications. | |
43 | ||
44 | CAT and Code Data Prioritization (CDP) features allow management of the CPU's | |
45 | last level cache. CAT introduces classes of service (COS) that are essentially | |
46 | bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to | |
47 | one cache way in last level cache. | |
48 | A CPU core is always assigned to one of the CAT classes. | |
49 | By programming CPU core assignment and COS bitmasks, applications can be given | |
50 | exclusive, shared, or mixed access to the CPU's last level cache. | |
51 | CDP extends CAT so that there are two bitmasks per COS, | |
52 | one for data and one for code. | |
53 | The number of classes and number of valid bits in a COS bitmask is CPU model | |
54 | specific and COS bitmasks need to be contiguous. Sample code calls this bitmask | |
55 | ``cbm`` or capacity bitmask. | |
56 | By default, after reset, all CPU cores are assigned to COS 0 and all classes | |
57 | are programmed to allow fill into all cache ways. | |
58 | CDP is off by default. | |
59 | ||
60 | For more information about CAT please see: | |
61 | ||
62 | * https://github.com/01org/intel-cmt-cat | |
63 | ||
64 | White paper demonstrating example use case: | |
65 | ||
66 | * `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_ | |
67 | ||
68 | Compiling the Application | |
69 | ------------------------- | |
70 | ||
71 | Requires ``libpqos`` from Intel's | |
72 | `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_ | |
73 | hosted on GitHub repository. For installation notes, please see ``README`` file. | |
74 | ||
75 | GIT: | |
76 | ||
77 | * https://github.com/01org/intel-cmt-cat | |
78 | ||
79 | To compile the application export the path to PQoS lib | |
80 | and the DPDK source tree and go to the example directory: | |
81 | ||
82 | .. code-block:: console | |
83 | ||
84 | export PQOS_INSTALL_PATH=/path/to/libpqos | |
85 | export RTE_SDK=/path/to/rte_sdk | |
86 | ||
87 | cd ${RTE_SDK}/examples/l2fwd-cat | |
88 | ||
89 | Set the target, for example: | |
90 | ||
91 | .. code-block:: console | |
92 | ||
93 | export RTE_TARGET=x86_64-native-linuxapp-gcc | |
94 | ||
95 | See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values. | |
96 | ||
97 | Build the application as follows: | |
98 | ||
99 | .. code-block:: console | |
100 | ||
101 | make | |
102 | ||
103 | ||
104 | Running the Application | |
105 | ----------------------- | |
106 | ||
107 | To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2: | |
108 | ||
109 | .. code-block:: console | |
110 | ||
111 | ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)" | |
112 | ||
113 | or to enable CAT and CDP on cpus 1,3: | |
114 | ||
115 | .. code-block:: console | |
116 | ||
117 | ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)" | |
118 | ||
119 | If CDP is not supported it will fail with following error message: | |
120 | ||
121 | .. code-block:: console | |
122 | ||
123 | PQOS: CDP requested but not supported. | |
124 | PQOS: Requested CAT configuration is not valid! | |
125 | PQOS: Shutting down PQoS library... | |
126 | EAL: Error - exiting with code: 1 | |
127 | Cause: PQOS: L3CA init failed! | |
128 | ||
129 | The option to enable CAT is: | |
130 | ||
131 | * ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``: | |
132 | ||
133 | where ``cbm`` stands for capacity bitmask and must be expressed in | |
134 | hexadecimal form. | |
135 | ||
136 | ``common_cbm`` is a single mask, for a CDP enabled system, a group of two | |
137 | masks (``code_cbm`` and ``data_cbm``) is used. | |
138 | ||
139 | ``(`` and ``)`` are necessary if it's a group. | |
140 | ||
141 | ``cpus`` could be a single digit/range or a group and must be expressed in | |
142 | decimal form. | |
143 | ||
144 | ``(`` and ``)`` are necessary if it's a group. | |
145 | ||
146 | e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'`` | |
147 | ||
148 | * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6; | |
149 | ||
150 | * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3; | |
151 | ||
152 | * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways; | |
153 | ||
154 | * cpu 7 has exclusive access to all of its 4 ways; | |
155 | ||
156 | e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system | |
157 | ||
158 | * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and | |
159 | data ways are not overlapping. | |
160 | ||
161 | ||
162 | Refer to *DPDK Getting Started Guide* for general information on running | |
163 | applications and the Environment Abstraction Layer (EAL) options. | |
164 | ||
165 | ||
166 | To reset or list CAT configuration and control CDP please use ``pqos`` tool | |
167 | from Intel's | |
168 | `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_. | |
169 | ||
170 | To enabled or disable CDP: | |
171 | ||
172 | .. code-block:: console | |
173 | ||
174 | sudo ./pqos -S cdp-on | |
175 | ||
176 | sudo ./pqos -S cdp-off | |
177 | ||
178 | to reset CAT configuration: | |
179 | ||
180 | .. code-block:: console | |
181 | ||
182 | sudo ./pqos -R | |
183 | ||
184 | to list CAT config: | |
185 | ||
186 | .. code-block:: console | |
187 | ||
188 | sudo ./pqos -s | |
189 | ||
190 | For more info about ``pqos`` tool please see its man page or | |
191 | `intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_. | |
192 | ||
193 | ||
194 | Explanation | |
195 | ----------- | |
196 | ||
197 | The following sections provide an explanation of the main components of the | |
198 | code. | |
199 | ||
200 | All DPDK library functions used in the sample code are prefixed with ``rte_`` | |
201 | and are explained in detail in the *DPDK API Documentation*. | |
202 | ||
203 | ||
204 | The Main Function | |
205 | ~~~~~~~~~~~~~~~~~ | |
206 | ||
207 | The ``main()`` function performs the initialization and calls the execution | |
208 | threads for each lcore. | |
209 | ||
210 | The first task is to initialize the Environment Abstraction Layer (EAL). The | |
211 | ``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` | |
212 | function. The value returned is the number of parsed arguments: | |
213 | ||
214 | .. code-block:: c | |
215 | ||
216 | int ret = rte_eal_init(argc, argv); | |
217 | if (ret < 0) | |
218 | rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); | |
219 | ||
220 | The next task is to initialize the PQoS library and configure CAT. The | |
221 | ``argc`` and ``argv`` arguments are provided to the ``cat_init()`` | |
222 | function. The value returned is the number of parsed arguments: | |
223 | ||
224 | .. code-block:: c | |
225 | ||
226 | int ret = cat_init(argc, argv); | |
227 | if (ret < 0) | |
228 | rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n"); | |
229 | ||
230 | ``cat_init()`` is a wrapper function which parses the command, validates | |
231 | the requested parameters and configures CAT accordingly. | |
232 | ||
233 | Parsing of command line arguments is done in ``parse_args(...)``. | |
234 | libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is | |
235 | queried for system CPU information and L3CA capabilities via | |
236 | ``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)`` | |
237 | calls. When all capability and topology information is collected, the requested | |
238 | CAT configuration is validated. A check is then performed (on per socket basis) | |
239 | for a sufficient number of un-associated COS. COS are selected and | |
240 | configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to | |
241 | relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls. | |
242 | ||
243 | ``atexit(...)`` is used to register ``cat_exit(...)`` to be called on | |
244 | a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating | |
245 | COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls. |