]>
Commit | Line | Data |
---|---|---|
9f95a23c TL |
1 | .. SPDX-License-Identifier: BSD-3-Clause |
2 | Copyright 2016 NXP | |
3 | ||
11fdf7f2 TL |
4 | |
5 | DPAA2 Poll Mode Driver | |
6 | ====================== | |
7 | ||
8 | The DPAA2 NIC PMD (**librte_pmd_dpaa2**) provides poll mode driver | |
9 | support for the inbuilt NIC found in the **NXP DPAA2** SoC family. | |
10 | ||
11 | More information can be found at `NXP Official Website | |
12 | <http://www.nxp.com/products/microcontrollers-and-processors/arm-processors/qoriq-arm-processors:QORIQ-ARM>`_. | |
13 | ||
14 | NXP DPAA2 (Data Path Acceleration Architecture Gen2) | |
15 | ---------------------------------------------------- | |
16 | ||
17 | This section provides an overview of the NXP DPAA2 architecture | |
18 | and how it is integrated into the DPDK. | |
19 | ||
20 | Contents summary | |
21 | ||
22 | - DPAA2 overview | |
23 | - Overview of DPAA2 objects | |
24 | - DPAA2 driver architecture overview | |
25 | ||
26 | .. _dpaa2_overview: | |
27 | ||
28 | DPAA2 Overview | |
29 | ~~~~~~~~~~~~~~ | |
30 | ||
31 | Reference: `FSL MC BUS in Linux Kernel <https://www.kernel.org/doc/readme/drivers-staging-fsl-mc-README.txt>`_. | |
32 | ||
33 | DPAA2 is a hardware architecture designed for high-speed network | |
34 | packet processing. DPAA2 consists of sophisticated mechanisms for | |
35 | processing Ethernet packets, queue management, buffer management, | |
36 | autonomous L2 switching, virtual Ethernet bridging, and accelerator | |
37 | (e.g. crypto) sharing. | |
38 | ||
39 | A DPAA2 hardware component called the Management Complex (or MC) manages the | |
40 | DPAA2 hardware resources. The MC provides an object-based abstraction for | |
41 | software drivers to use the DPAA2 hardware. | |
42 | ||
43 | The MC uses DPAA2 hardware resources such as queues, buffer pools, and | |
44 | network ports to create functional objects/devices such as network | |
45 | interfaces, an L2 switch, or accelerator instances. | |
46 | ||
47 | The MC provides memory-mapped I/O command interfaces (MC portals) | |
48 | which DPAA2 software drivers use to operate on DPAA2 objects: | |
49 | ||
50 | The diagram below shows an overview of the DPAA2 resource management | |
51 | architecture: | |
52 | ||
53 | .. code-block:: console | |
54 | ||
55 | +--------------------------------------+ | |
56 | | OS | | |
57 | | DPAA2 drivers | | |
58 | | | | | |
59 | +-----------------------------|--------+ | |
60 | | | |
61 | | (create,discover,connect | |
62 | | config,use,destroy) | |
63 | | | |
64 | DPAA2 | | |
65 | +------------------------| mc portal |-+ | |
66 | | | | | |
67 | | +- - - - - - - - - - - - -V- - -+ | | |
68 | | | | | | |
69 | | | Management Complex (MC) | | | |
70 | | | | | | |
71 | | +- - - - - - - - - - - - - - - -+ | | |
72 | | | | |
73 | | Hardware Hardware | | |
74 | | Resources Objects | | |
75 | | --------- ------- | | |
76 | | -queues -DPRC | | |
77 | | -buffer pools -DPMCP | | |
78 | | -Eth MACs/ports -DPIO | | |
79 | | -network interface -DPNI | | |
80 | | profiles -DPMAC | | |
81 | | -queue portals -DPBP | | |
82 | | -MC portals ... | | |
83 | | ... | | |
84 | | | | |
85 | +--------------------------------------+ | |
86 | ||
87 | The MC mediates operations such as create, discover, | |
88 | connect, configuration, and destroy. Fast-path operations | |
89 | on data, such as packet transmit/receive, are not mediated by | |
90 | the MC and are done directly using memory mapped regions in | |
91 | DPIO objects. | |
92 | ||
93 | Overview of DPAA2 Objects | |
94 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | |
95 | ||
96 | The section provides a brief overview of some key DPAA2 objects. | |
97 | A simple scenario is described illustrating the objects involved | |
98 | in creating a network interfaces. | |
99 | ||
100 | DPRC (Datapath Resource Container) | |
101 | ||
102 | A DPRC is a container object that holds all the other | |
103 | types of DPAA2 objects. In the example diagram below there | |
104 | are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) | |
105 | in the container. | |
106 | ||
107 | .. code-block:: console | |
108 | ||
109 | +---------------------------------------------------------+ | |
110 | | DPRC | | |
111 | | | | |
112 | | +-------+ +-------+ +-------+ +-------+ +-------+ | | |
113 | | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | | |
114 | | +-------+ +-------+ +-------+ +---+---+ +---+---+ | | |
115 | | | DPMCP | | DPIO | | | |
116 | | +-------+ +-------+ | | |
117 | | | DPMCP | | | |
118 | | +-------+ | | |
119 | | | | |
120 | +---------------------------------------------------------+ | |
121 | ||
122 | From the point of view of an OS, a DPRC behaves similar to a plug and | |
123 | play bus, like PCI. DPRC commands can be used to enumerate the contents | |
124 | of the DPRC, discover the hardware objects present (including mappable | |
125 | regions and interrupts). | |
126 | ||
127 | .. code-block:: console | |
128 | ||
129 | DPRC.1 (bus) | |
130 | | | |
131 | +--+--------+-------+-------+-------+ | |
132 | | | | | | | |
133 | DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1 | |
134 | DPMCP.2 DPIO.2 | |
135 | DPMCP.3 | |
136 | ||
137 | Hardware objects can be created and destroyed dynamically, providing | |
138 | the ability to hot plug/unplug objects in and out of the DPRC. | |
139 | ||
140 | A DPRC has a mappable MMIO region (an MC portal) that can be used | |
141 | to send MC commands. It has an interrupt for status events (like | |
142 | hotplug). | |
143 | ||
144 | All objects in a container share the same hardware "isolation context". | |
145 | This means that with respect to an IOMMU the isolation granularity | |
146 | is at the DPRC (container) level, not at the individual object | |
147 | level. | |
148 | ||
149 | DPRCs can be defined statically and populated with objects | |
150 | via a config file passed to the MC when firmware starts | |
151 | it. There is also a Linux user space tool called "restool" | |
152 | that can be used to create/destroy containers and objects | |
153 | dynamically. | |
154 | ||
155 | DPAA2 Objects for an Ethernet Network Interface | |
156 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
157 | ||
158 | A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX | |
159 | queuing mechanisms, configuration mechanisms, buffer management, | |
160 | physical ports, and interrupts. DPAA2 uses a more granular approach | |
161 | utilizing multiple hardware objects. Each object provides specialized | |
162 | functions. Groups of these objects are used by software to provide | |
163 | Ethernet network interface functionality. This approach provides | |
164 | efficient use of finite hardware resources, flexibility, and | |
165 | performance advantages. | |
166 | ||
167 | The diagram below shows the objects needed for a simple | |
168 | network interface configuration on a system with 2 CPUs. | |
169 | ||
170 | .. code-block:: console | |
171 | ||
172 | +---+---+ +---+---+ | |
173 | CPU0 CPU1 | |
174 | +---+---+ +---+---+ | |
175 | | | | |
176 | +---+---+ +---+---+ | |
177 | DPIO DPIO | |
178 | +---+---+ +---+---+ | |
179 | \ / | |
180 | \ / | |
181 | \ / | |
182 | +---+---+ | |
183 | DPNI --- DPBP,DPMCP | |
184 | +---+---+ | |
185 | | | |
186 | | | |
187 | +---+---+ | |
188 | DPMAC | |
189 | +---+---+ | |
190 | | | |
191 | port/PHY | |
192 | ||
193 | Below the objects are described. For each object a brief description | |
194 | is provided along with a summary of the kinds of operations the object | |
195 | supports and a summary of key resources of the object (MMIO regions | |
196 | and IRQs). | |
197 | ||
198 | DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a | |
199 | hardware device that connects to an Ethernet PHY and allows | |
200 | physical transmission and reception of Ethernet frames. | |
201 | ||
202 | - MMIO regions: none | |
203 | - IRQs: DPNI link change | |
204 | - commands: set link up/down, link config, get stats, IRQ config, enable, reset | |
205 | ||
206 | DPNI (Datapath Network Interface): contains TX/RX queues, | |
207 | network interface configuration, and RX buffer pool configuration | |
208 | mechanisms. The TX/RX queues are in memory and are identified by | |
209 | queue number. | |
210 | ||
211 | - MMIO regions: none | |
212 | - IRQs: link state | |
213 | - commands: port config, offload config, queue config, parse/classify config, IRQ config, enable, reset | |
214 | ||
215 | DPIO (Datapath I/O): provides interfaces to enqueue and dequeue | |
216 | packets and do hardware buffer pool management operations. The DPAA2 | |
217 | architecture separates the mechanism to access queues (the DPIO object) | |
218 | from the queues themselves. The DPIO provides an MMIO interface to | |
219 | enqueue/dequeue packets. To enqueue something a descriptor is written | |
220 | to the DPIO MMIO region, which includes the target queue number. | |
221 | There will typically be one DPIO assigned to each CPU. This allows all | |
222 | CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are | |
223 | expected to be shared by different DPAA2 drivers. | |
224 | ||
225 | - MMIO regions: queue operations, buffer management | |
226 | - IRQs: data availability, congestion notification, buffer pool depletion | |
227 | - commands: IRQ config, enable, reset | |
228 | ||
229 | DPBP (Datapath Buffer Pool): represents a hardware buffer | |
230 | pool. | |
231 | ||
232 | - MMIO regions: none | |
233 | - IRQs: none | |
234 | - commands: enable, reset | |
235 | ||
236 | DPMCP (Datapath MC Portal): provides an MC command portal. | |
237 | Used by drivers to send commands to the MC to manage | |
238 | objects. | |
239 | ||
240 | - MMIO regions: MC command portal | |
241 | - IRQs: command completion | |
242 | - commands: IRQ config, enable, reset | |
243 | ||
244 | Object Connections | |
245 | ~~~~~~~~~~~~~~~~~~ | |
246 | ||
247 | Some objects have explicit relationships that must | |
248 | be configured: | |
249 | ||
250 | - DPNI <--> DPMAC | |
251 | - DPNI <--> DPNI | |
252 | - DPNI <--> L2-switch-port | |
253 | ||
254 | A DPNI must be connected to something such as a DPMAC, | |
255 | another DPNI, or L2 switch port. The DPNI connection | |
256 | is made via a DPRC command. | |
257 | ||
258 | .. code-block:: console | |
259 | ||
260 | +-------+ +-------+ | |
261 | | DPNI | | DPMAC | | |
262 | +---+---+ +---+---+ | |
263 | | | | |
264 | +==========+ | |
265 | ||
266 | - DPNI <--> DPBP | |
267 | ||
268 | A network interface requires a 'buffer pool' (DPBP object) which provides | |
269 | a list of pointers to memory where received Ethernet data is to be copied. | |
270 | The Ethernet driver configures the DPBPs associated with the network | |
271 | interface. | |
272 | ||
273 | Interrupts | |
274 | ~~~~~~~~~~ | |
275 | ||
276 | All interrupts generated by DPAA2 objects are message | |
277 | interrupts. At the hardware level message interrupts | |
278 | generated by devices will normally have 3 components-- | |
279 | 1) a non-spoofable 'device-id' expressed on the hardware | |
280 | bus, 2) an address, 3) a data value. | |
281 | ||
282 | In the case of DPAA2 devices/objects, all objects in the | |
283 | same container/DPRC share the same 'device-id'. | |
284 | For ARM-based SoC this is the same as the stream ID. | |
285 | ||
286 | ||
287 | DPAA2 DPDK - Poll Mode Driver Overview | |
288 | -------------------------------------- | |
289 | ||
290 | This section provides an overview of the drivers for | |
291 | DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" | |
292 | drivers and 2) functional object drivers (such as Ethernet). | |
293 | ||
294 | As described previously, a DPRC is a container that holds the other | |
295 | types of DPAA2 objects. It is functionally similar to a plug-and-play | |
296 | bus controller. | |
297 | ||
298 | Each object in the DPRC is a Linux "device" and is bound to a driver. | |
299 | The diagram below shows the dpaa2 drivers involved in a networking | |
300 | scenario and the objects bound to each driver. A brief description | |
301 | of each driver follows. | |
302 | ||
303 | .. code-block: console | |
304 | ||
305 | ||
306 | +------------+ | |
307 | | DPDK DPAA2 | | |
308 | | PMD | | |
309 | +------------+ +------------+ | |
310 | | Ethernet |.......| Mempool | | |
311 | . . . . . . . . . | (DPNI) | | (DPBP) | | |
312 | . +---+---+----+ +-----+------+ | |
313 | . ^ | . | |
314 | . | |<enqueue, . | |
315 | . | | dequeue> . | |
316 | . | | . | |
317 | . +---+---V----+ . | |
318 | . . . . . . . . . . .| DPIO driver| . | |
319 | . . | (DPIO) | . | |
320 | . . +-----+------+ . | |
321 | . . | QBMAN | . | |
322 | . . | Driver | . | |
323 | +----+------+-------+ +-----+----- | . | |
324 | | dpaa2 bus | | . | |
325 | | VFIO fslmc-bus |....................|..................... | |
326 | | | | | |
327 | | /bus/fslmc | | | |
328 | +-------------------+ | | |
329 | | | |
330 | ========================== HARDWARE =====|======================= | |
331 | DPIO | |
332 | | | |
333 | DPNI---DPBP | |
334 | | | |
335 | DPMAC | |
336 | | | |
337 | PHY | |
338 | =========================================|======================== | |
339 | ||
340 | ||
341 | A brief description of each driver is provided below. | |
342 | ||
343 | DPAA2 bus driver | |
344 | ~~~~~~~~~~~~~~~~ | |
345 | ||
346 | The DPAA2 bus driver is a rte_bus driver which scans the fsl-mc bus. | |
347 | Key functions include: | |
348 | ||
349 | - Reading the container and setting up vfio group | |
350 | - Scanning and parsing the various MC objects and adding them to | |
351 | their respective device list. | |
352 | ||
353 | Additionally, it also provides the object driver for generic MC objects. | |
354 | ||
355 | DPIO driver | |
356 | ~~~~~~~~~~~ | |
357 | ||
358 | The DPIO driver is bound to DPIO objects and provides services that allow | |
359 | other drivers such as the Ethernet driver to enqueue and dequeue data for | |
360 | their respective objects. | |
361 | Key services include: | |
362 | ||
363 | - Data availability notifications | |
364 | - Hardware queuing operations (enqueue and dequeue of data) | |
365 | - Hardware buffer pool management | |
366 | ||
367 | To transmit a packet the Ethernet driver puts data on a queue and | |
368 | invokes a DPIO API. For receive, the Ethernet driver registers | |
369 | a data availability notification callback. To dequeue a packet | |
370 | a DPIO API is used. | |
371 | ||
372 | There is typically one DPIO object per physical CPU for optimum | |
373 | performance, allowing different CPUs to simultaneously enqueue | |
374 | and dequeue data. | |
375 | ||
376 | The DPIO driver operates on behalf of all DPAA2 drivers | |
377 | active -- Ethernet, crypto, compression, etc. | |
378 | ||
379 | DPBP based Mempool driver | |
380 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | |
381 | ||
9f95a23c | 382 | The DPBP driver is bound to a DPBP objects and provides services to |
11fdf7f2 TL |
383 | create a hardware offloaded packet buffer mempool. |
384 | ||
385 | DPAA2 NIC Driver | |
386 | ~~~~~~~~~~~~~~~~ | |
387 | The Ethernet driver is bound to a DPNI and implements the kernel | |
388 | interfaces needed to connect the DPAA2 network interface to | |
389 | the network stack. | |
390 | ||
391 | Each DPNI corresponds to a DPDK network interface. | |
392 | ||
393 | Features | |
394 | ^^^^^^^^ | |
395 | ||
396 | Features of the DPAA2 PMD are: | |
397 | ||
398 | - Multiple queues for TX and RX | |
399 | - Receive Side Scaling (RSS) | |
9f95a23c | 400 | - MAC/VLAN filtering |
11fdf7f2 TL |
401 | - Packet type information |
402 | - Checksum offload | |
403 | - Promiscuous mode | |
9f95a23c TL |
404 | - Multicast mode |
405 | - Port hardware statistics | |
406 | - Jumbo frames | |
407 | - Link flow control | |
408 | - Scattered and gather for TX and RX | |
11fdf7f2 TL |
409 | |
410 | Supported DPAA2 SoCs | |
411 | -------------------- | |
9f95a23c | 412 | - LX2160A |
11fdf7f2 TL |
413 | - LS2084A/LS2044A |
414 | - LS2088A/LS2048A | |
415 | - LS1088A/LS1048A | |
416 | ||
417 | Prerequisites | |
418 | ------------- | |
419 | ||
9f95a23c | 420 | See :doc:`../platform/dpaa2` for setup information |
11fdf7f2 TL |
421 | |
422 | Currently supported by DPDK: | |
423 | ||
9f95a23c TL |
424 | - NXP LSDK **19.03+**. |
425 | - MC Firmware version **10.14.0** and higher. | |
11fdf7f2 TL |
426 | - Supported architectures: **arm64 LE**. |
427 | ||
428 | - Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment. | |
429 | ||
430 | .. note:: | |
431 | ||
432 | Some part of fslmc bus code (mc flib - object library) routines are | |
9f95a23c | 433 | dual licensed (BSD & GPLv2), however they are used as BSD in DPDK in userspace. |
11fdf7f2 TL |
434 | |
435 | Pre-Installation Configuration | |
436 | ------------------------------ | |
437 | ||
438 | Config File Options | |
439 | ~~~~~~~~~~~~~~~~~~~ | |
440 | ||
441 | The following options can be modified in the ``config`` file. | |
442 | Please note that enabling debugging options may affect system performance. | |
443 | ||
444 | - ``CONFIG_RTE_LIBRTE_FSLMC_BUS`` (default ``n``) | |
445 | ||
446 | By default it is enabled only for defconfig_arm64-dpaa2-* config. | |
447 | Toggle compilation of the ``librte_bus_fslmc`` driver. | |
448 | ||
449 | - ``CONFIG_RTE_LIBRTE_DPAA2_PMD`` (default ``n``) | |
450 | ||
451 | By default it is enabled only for defconfig_arm64-dpaa2-* config. | |
452 | Toggle compilation of the ``librte_pmd_dpaa2`` driver. | |
453 | ||
454 | - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_DRIVER`` (default ``n``) | |
455 | ||
9f95a23c | 456 | Toggle display of debugging messages/logic |
11fdf7f2 TL |
457 | |
458 | - ``CONFIG_RTE_LIBRTE_DPAA2_USE_PHYS_IOVA`` (default ``y``) | |
459 | ||
460 | Toggle to use physical address vs virtual address for hardware accelerators. | |
461 | ||
11fdf7f2 TL |
462 | Driver compilation and testing |
463 | ------------------------------ | |
464 | ||
465 | Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` | |
466 | for details. | |
467 | ||
468 | #. Running testpmd: | |
469 | ||
470 | Follow instructions available in the document | |
471 | :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` | |
472 | to run testpmd. | |
473 | ||
474 | Example output: | |
475 | ||
476 | .. code-block:: console | |
477 | ||
9f95a23c | 478 | ./testpmd -c 0xff -n 1 -- -i --portmask=0x3 --nb-cores=1 --no-flush-rx |
11fdf7f2 TL |
479 | |
480 | ..... | |
481 | EAL: Registered [pci] bus. | |
482 | EAL: Registered [fslmc] bus. | |
483 | EAL: Detected 8 lcore(s) | |
484 | EAL: Probing VFIO support... | |
485 | EAL: VFIO support initialized | |
486 | ..... | |
487 | PMD: DPAA2: Processing Container = dprc.2 | |
488 | EAL: fslmc: DPRC contains = 51 devices | |
489 | EAL: fslmc: Bus scan completed | |
490 | ..... | |
491 | Configuring Port 0 (socket 0) | |
492 | Port 0: 00:00:00:00:00:01 | |
493 | Configuring Port 1 (socket 0) | |
494 | Port 1: 00:00:00:00:00:02 | |
495 | ..... | |
496 | Checking link statuses... | |
497 | Port 0 Link Up - speed 10000 Mbps - full-duplex | |
498 | Port 1 Link Up - speed 10000 Mbps - full-duplex | |
499 | Done | |
500 | testpmd> | |
501 | ||
9f95a23c TL |
502 | |
503 | * Use dev arg option ``drv_loopback=1`` to loopback packets at | |
504 | driver level. Any packet received will be reflected back by the | |
505 | driver on same port. e.g. ``fslmc:dpni.1,drv_loopback=1`` | |
506 | ||
507 | Enabling logs | |
508 | ------------- | |
509 | ||
510 | For enabling logging for DPAA2 PMD, following log-level prefix can be used: | |
511 | ||
512 | .. code-block:: console | |
513 | ||
514 | <dpdk app> <EAL args> --log-level=bus.fslmc:<level> -- ... | |
515 | ||
516 | Using ``bus.fslmc`` as log matching criteria, all FSLMC bus logs can be enabled | |
517 | which are lower than logging ``level``. | |
518 | ||
519 | Or | |
520 | ||
521 | .. code-block:: console | |
522 | ||
523 | <dpdk app> <EAL args> --log-level=pmd.net.dpaa2:<level> -- ... | |
524 | ||
525 | Using ``pmd.net.dpaa2`` as log matching criteria, all PMD logs can be enabled | |
526 | which are lower than logging ``level``. | |
527 | ||
528 | Whitelisting & Blacklisting | |
529 | --------------------------- | |
530 | ||
531 | For blacklisting a DPAA2 device, following commands can be used. | |
532 | ||
533 | .. code-block:: console | |
534 | ||
535 | <dpdk app> <EAL args> -b "fslmc:dpni.x" -- ... | |
536 | ||
537 | Where x is the device object id as configured in resource container. | |
538 | ||
11fdf7f2 TL |
539 | Limitations |
540 | ----------- | |
541 | ||
542 | Platform Requirement | |
543 | ~~~~~~~~~~~~~~~~~~~~ | |
544 | DPAA2 drivers for DPDK can only work on NXP SoCs as listed in the | |
545 | ``Supported DPAA2 SoCs``. | |
546 | ||
547 | Maximum packet length | |
548 | ~~~~~~~~~~~~~~~~~~~~~ | |
549 | ||
550 | The DPAA2 SoC family support a maximum of a 10240 jumbo frame. The value | |
551 | is fixed and cannot be changed. So, even when the ``rxmode.max_rx_pkt_len`` | |
552 | member of ``struct rte_eth_conf`` is set to a value lower than 10240, frames | |
553 | up to 10240 bytes can still reach the host interface. | |
9f95a23c TL |
554 | |
555 | Other Limitations | |
556 | ~~~~~~~~~~~~~~~~~ | |
557 | ||
558 | - RSS hash key cannot be modified. | |
559 | - RSS RETA cannot be configured. |