]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | .. BSD LICENSE |
2 | Copyright 2015 Chelsio Communications. | |
3 | All rights reserved. | |
4 | ||
5 | Redistribution and use in source and binary forms, with or without | |
6 | modification, are permitted provided that the following conditions | |
7 | are met: | |
8 | ||
9 | * Redistributions of source code must retain the above copyright | |
10 | notice, this list of conditions and the following disclaimer. | |
11 | * Redistributions in binary form must reproduce the above copyright | |
12 | notice, this list of conditions and the following disclaimer in | |
13 | the documentation and/or other materials provided with the | |
14 | distribution. | |
15 | * Neither the name of Chelsio Communications nor the names of its | |
16 | contributors may be used to endorse or promote products derived | |
17 | from this software without specific prior written permission. | |
18 | ||
19 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | |
20 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | |
21 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | |
22 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | |
23 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | |
24 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | |
25 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | |
26 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | |
27 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | |
28 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | |
29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
30 | ||
31 | CXGBE Poll Mode Driver | |
32 | ====================== | |
33 | ||
34 | The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support | |
35 | for **Chelsio T5** 10/40 Gbps family of adapters. CXGBE PMD has support | |
36 | for the latest Linux and FreeBSD operating systems. | |
37 | ||
38 | More information can be found at `Chelsio Communications Official Website | |
39 | <http://www.chelsio.com>`_. | |
40 | ||
41 | Features | |
42 | -------- | |
43 | ||
44 | CXGBE PMD has support for: | |
45 | ||
46 | - Multiple queues for TX and RX | |
47 | - Receiver Side Steering (RSS) | |
48 | - VLAN filtering | |
49 | - Checksum offload | |
50 | - Promiscuous mode | |
51 | - All multicast mode | |
52 | - Port hardware statistics | |
53 | - Jumbo frames | |
54 | ||
55 | Limitations | |
56 | ----------- | |
57 | ||
58 | The Chelsio T5 devices provide two/four ports but expose a single PCI bus | |
59 | address, thus, librte_pmd_cxgbe registers itself as a | |
60 | PCI driver that allocates one Ethernet device per detected port. | |
61 | ||
62 | For this reason, one cannot whitelist/blacklist a single port without | |
63 | whitelisting/blacklisting the other ports on the same device. | |
64 | ||
65 | Supported Chelsio T5 NICs | |
66 | ------------------------- | |
67 | ||
68 | - 1G NICs: T502-BT | |
69 | - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR | |
70 | - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR | |
71 | - Other T5 NICs: T522-CR | |
72 | ||
73 | Prerequisites | |
74 | ------------- | |
75 | ||
76 | - Requires firmware version **1.13.32.0** and higher. Visit | |
77 | `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware | |
78 | bundled with the latest Chelsio Unified Wire package. | |
79 | ||
80 | For Linux, installing and loading the latest cxgb4 kernel driver from the | |
81 | Chelsio Unified Wire package should get you the latest firmware. More | |
82 | information can be obtained from the User Guide that is bundled with the | |
83 | Chelsio Unified Wire package. | |
84 | ||
85 | For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire | |
86 | package must be manually flashed via cxgbetool available in FreeBSD source | |
87 | repository. | |
88 | ||
89 | Instructions on how to manually flash the firmware are given in section | |
90 | :ref:`linux-installation` for Linux and section :ref:`freebsd-installation` | |
91 | for FreeBSD. | |
92 | ||
93 | Pre-Installation Configuration | |
94 | ------------------------------ | |
95 | ||
96 | Config File Options | |
97 | ~~~~~~~~~~~~~~~~~~~ | |
98 | ||
99 | The following options can be modified in the ``.config`` file. Please note that | |
100 | enabling debugging options may affect system performance. | |
101 | ||
102 | - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**) | |
103 | ||
104 | Toggle compilation of librte_pmd_cxgbe driver. | |
105 | ||
106 | - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**) | |
107 | ||
108 | Toggle display of generic debugging messages. | |
109 | ||
110 | - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**) | |
111 | ||
112 | Toggle display of registers related run-time check messages. | |
113 | ||
114 | - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**) | |
115 | ||
116 | Toggle display of firmware mailbox related run-time check messages. | |
117 | ||
118 | - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**) | |
119 | ||
120 | Toggle display of transmission data path run-time check messages. | |
121 | ||
122 | - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**) | |
123 | ||
124 | Toggle display of receiving data path run-time check messages. | |
125 | ||
126 | .. _driver-compilation: | |
127 | ||
128 | Driver Compilation | |
129 | ~~~~~~~~~~~~~~~~~~ | |
130 | ||
131 | To compile CXGBE PMD for Linux x86_64 gcc target, run the following "make" | |
132 | command: | |
133 | ||
134 | .. code-block:: console | |
135 | ||
136 | cd <DPDK-source-directory> | |
137 | make config T=x86_64-native-linuxapp-gcc install | |
138 | ||
139 | To compile CXGBE PMD for FreeBSD x86_64 clang target, run the following "gmake" | |
140 | command: | |
141 | ||
142 | .. code-block:: console | |
143 | ||
144 | cd <DPDK-source-directory> | |
145 | gmake config T=x86_64-native-bsdapp-clang install | |
146 | ||
147 | Linux | |
148 | ----- | |
149 | ||
150 | .. _linux-installation: | |
151 | ||
152 | Linux Installation | |
153 | ~~~~~~~~~~~~~~~~~~ | |
154 | ||
155 | Steps to manually install the latest firmware from the downloaded Chelsio | |
156 | Unified Wire package for Linux operating system are as follows: | |
157 | ||
158 | #. Load the kernel module: | |
159 | ||
160 | .. code-block:: console | |
161 | ||
162 | modprobe cxgb4 | |
163 | ||
164 | #. Use ifconfig to get the interface name assigned to Chelsio card: | |
165 | ||
166 | .. code-block:: console | |
167 | ||
168 | ifconfig -a | grep "00:07:43" | |
169 | ||
170 | Example output: | |
171 | ||
172 | .. code-block:: console | |
173 | ||
174 | p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0 | |
175 | p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8 | |
176 | ||
177 | #. Install cxgbtool: | |
178 | ||
179 | .. code-block:: console | |
180 | ||
181 | cd <path_to_uwire>/tools/cxgbtool | |
182 | make install | |
183 | ||
184 | #. Use cxgbtool to load the firmware config file onto the card: | |
185 | ||
186 | .. code-block:: console | |
187 | ||
188 | cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt | |
189 | ||
190 | #. Use cxgbtool to load the firmware image onto the card: | |
191 | ||
192 | .. code-block:: console | |
193 | ||
194 | cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin | |
195 | ||
196 | #. Unload and reload the kernel module: | |
197 | ||
198 | .. code-block:: console | |
199 | ||
200 | modprobe -r cxgb4 | |
201 | modprobe cxgb4 | |
202 | ||
203 | #. Verify with ethtool: | |
204 | ||
205 | .. code-block:: console | |
206 | ||
207 | ethtool -i p1p1 | grep "firmware" | |
208 | ||
209 | Example output: | |
210 | ||
211 | .. code-block:: console | |
212 | ||
213 | firmware-version: 1.13.32.0, TP 0.1.4.8 | |
214 | ||
215 | Running testpmd | |
216 | ~~~~~~~~~~~~~~~ | |
217 | ||
218 | This section demonstrates how to launch **testpmd** with Chelsio T5 | |
219 | devices managed by librte_pmd_cxgbe in Linux operating system. | |
220 | ||
221 | #. Change to DPDK source directory where the target has been compiled in | |
222 | section :ref:`driver-compilation`: | |
223 | ||
224 | .. code-block:: console | |
225 | ||
226 | cd <DPDK-source-directory> | |
227 | ||
228 | #. Load the kernel module: | |
229 | ||
230 | .. code-block:: console | |
231 | ||
232 | modprobe cxgb4 | |
233 | ||
234 | #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver: | |
235 | ||
236 | .. code-block:: console | |
237 | ||
238 | dmesg | tail -2 | |
239 | ||
240 | Example output: | |
241 | ||
242 | .. code-block:: console | |
243 | ||
244 | cxgb4 0000:02:00.4 p1p1: renamed from eth0 | |
245 | cxgb4 0000:02:00.4 p1p2: renamed from eth1 | |
246 | ||
247 | .. note:: | |
248 | ||
249 | Both the interfaces of a Chelsio T5 2-port adapter are bound to the | |
250 | same PCI bus address. | |
251 | ||
252 | #. Unload the kernel module: | |
253 | ||
254 | .. code-block:: console | |
255 | ||
256 | modprobe -ar cxgb4 csiostor | |
257 | ||
258 | #. Request huge pages: | |
259 | ||
260 | .. code-block:: console | |
261 | ||
262 | echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages | |
263 | ||
264 | #. Mount huge pages: | |
265 | ||
266 | .. code-block:: console | |
267 | ||
268 | mkdir /mnt/huge | |
269 | mount -t hugetlbfs nodev /mnt/huge | |
270 | ||
271 | #. Load igb_uio or vfio-pci driver: | |
272 | ||
273 | .. code-block:: console | |
274 | ||
275 | insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko | |
276 | ||
277 | or | |
278 | ||
279 | .. code-block:: console | |
280 | ||
281 | modprobe vfio-pci | |
282 | ||
283 | #. Bind the Chelsio T5 adapters to igb_uio or vfio-pci loaded in the previous | |
284 | step: | |
285 | ||
286 | .. code-block:: console | |
287 | ||
288 | ./tools/dpdk-devbind.py --bind igb_uio 0000:02:00.4 | |
289 | ||
290 | or | |
291 | ||
292 | Setup VFIO permissions for regular users and then bind to vfio-pci: | |
293 | ||
294 | .. code-block:: console | |
295 | ||
296 | sudo chmod a+x /dev/vfio | |
297 | ||
298 | sudo chmod 0666 /dev/vfio/* | |
299 | ||
300 | ./tools/dpdk-devbind.py --bind vfio-pci 0000:02:00.4 | |
301 | ||
302 | .. note:: | |
303 | ||
304 | Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs. | |
305 | ||
306 | #. Start testpmd with basic parameters: | |
307 | ||
308 | .. code-block:: console | |
309 | ||
310 | ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i | |
311 | ||
312 | Example output: | |
313 | ||
314 | .. code-block:: console | |
315 | ||
316 | [...] | |
317 | EAL: PCI device 0000:02:00.4 on NUMA socket -1 | |
318 | EAL: probe driver: 1425:5401 rte_cxgbe_pmd | |
319 | EAL: PCI memory mapped at 0x7fd7c0200000 | |
320 | EAL: PCI memory mapped at 0x7fd77cdfd000 | |
321 | EAL: PCI memory mapped at 0x7fd7c10b7000 | |
322 | PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8 | |
323 | PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter | |
324 | Interactive-mode selected | |
325 | Configuring Port 0 (socket 0) | |
326 | Port 0: 00:07:43:2D:EA:C0 | |
327 | Configuring Port 1 (socket 0) | |
328 | Port 1: 00:07:43:2D:EA:C8 | |
329 | Checking link statuses... | |
330 | PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted | |
331 | PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted | |
332 | Port 0 Link Up - speed 10000 Mbps - full-duplex | |
333 | Port 1 Link Up - speed 10000 Mbps - full-duplex | |
334 | Done | |
335 | testpmd> | |
336 | ||
337 | .. note:: | |
338 | ||
339 | Flow control pause TX/RX is disabled by default and can be enabled via | |
340 | testpmd. Refer section :ref:`flow-control` for more details. | |
341 | ||
342 | FreeBSD | |
343 | ------- | |
344 | ||
345 | .. _freebsd-installation: | |
346 | ||
347 | FreeBSD Installation | |
348 | ~~~~~~~~~~~~~~~~~~~~ | |
349 | ||
350 | Steps to manually install the latest firmware from the downloaded Chelsio | |
351 | Unified Wire package for FreeBSD operating system are as follows: | |
352 | ||
353 | #. Load the kernel module: | |
354 | ||
355 | .. code-block:: console | |
356 | ||
357 | kldload if_cxgbe | |
358 | ||
359 | #. Use dmesg to get the t5nex instance assigned to the Chelsio card: | |
360 | ||
361 | .. code-block:: console | |
362 | ||
363 | dmesg | grep "t5nex" | |
364 | ||
365 | Example output: | |
366 | ||
367 | .. code-block:: console | |
368 | ||
369 | t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2 | |
370 | cxl0: <port 0> on t5nex0 | |
371 | cxl1: <port 1> on t5nex0 | |
372 | t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq | |
373 | ||
374 | In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance. | |
375 | ||
376 | #. Install cxgbetool from FreeBSD source repository: | |
377 | ||
378 | .. code-block:: console | |
379 | ||
380 | cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/ | |
381 | make && make install | |
382 | ||
383 | #. Use cxgbetool to load the firmware image onto the card: | |
384 | ||
385 | .. code-block:: console | |
386 | ||
387 | cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin | |
388 | ||
389 | #. Unload and reload the kernel module: | |
390 | ||
391 | .. code-block:: console | |
392 | ||
393 | kldunload if_cxgbe | |
394 | kldload if_cxgbe | |
395 | ||
396 | #. Verify with sysctl: | |
397 | ||
398 | .. code-block:: console | |
399 | ||
400 | sysctl -a | grep "t5nex" | grep "firmware" | |
401 | ||
402 | Example output: | |
403 | ||
404 | .. code-block:: console | |
405 | ||
406 | dev.t5nex.0.firmware_version: 1.13.32.0 | |
407 | ||
408 | Running testpmd | |
409 | ~~~~~~~~~~~~~~~ | |
410 | ||
411 | This section demonstrates how to launch **testpmd** with Chelsio T5 | |
412 | devices managed by librte_pmd_cxgbe in FreeBSD operating system. | |
413 | ||
414 | #. Change to DPDK source directory where the target has been compiled in | |
415 | section :ref:`driver-compilation`: | |
416 | ||
417 | .. code-block:: console | |
418 | ||
419 | cd <DPDK-source-directory> | |
420 | ||
421 | #. Copy the contigmem kernel module to /boot/kernel directory: | |
422 | ||
423 | .. code-block:: console | |
424 | ||
425 | cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/ | |
426 | ||
427 | #. Add the following lines to /boot/loader.conf: | |
428 | ||
429 | .. code-block:: console | |
430 | ||
431 | # reserve 2 x 1G blocks of contiguous memory using contigmem driver | |
432 | hw.contigmem.num_buffers=2 | |
433 | hw.contigmem.buffer_size=1073741824 | |
434 | # load contigmem module during boot process | |
435 | contigmem_load="YES" | |
436 | ||
437 | The above lines load the contigmem kernel module during boot process and | |
438 | allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on. | |
439 | This is to avoid issues with potential memory fragmentation during later | |
440 | system up time, which may result in failure of allocating the contiguous | |
441 | memory required for the contigmem kernel module. | |
442 | ||
443 | #. Restart the system and ensure the contigmem module is loaded successfully: | |
444 | ||
445 | .. code-block:: console | |
446 | ||
447 | reboot | |
448 | kldstat | grep "contigmem" | |
449 | ||
450 | Example output: | |
451 | ||
452 | .. code-block:: console | |
453 | ||
454 | 2 1 0xffffffff817f1000 3118 contigmem.ko | |
455 | ||
456 | #. Repeat step 1 to ensure that you are in the DPDK source directory. | |
457 | ||
458 | #. Load the cxgbe kernel module: | |
459 | ||
460 | .. code-block:: console | |
461 | ||
462 | kldload if_cxgbe | |
463 | ||
464 | #. Get the PCI bus addresses of the interfaces bound to t5nex driver: | |
465 | ||
466 | .. code-block:: console | |
467 | ||
468 | pciconf -l | grep "t5nex" | |
469 | ||
470 | Example output: | |
471 | ||
472 | .. code-block:: console | |
473 | ||
474 | t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00 | |
475 | ||
476 | In the above example, the t5nex0 is bound to 2:0:4 bus address. | |
477 | ||
478 | .. note:: | |
479 | ||
480 | Both the interfaces of a Chelsio T5 2-port adapter are bound to the | |
481 | same PCI bus address. | |
482 | ||
483 | #. Unload the kernel module: | |
484 | ||
485 | .. code-block:: console | |
486 | ||
487 | kldunload if_cxgbe | |
488 | ||
489 | #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter: | |
490 | ||
491 | .. code-block:: console | |
492 | ||
493 | kenv hw.nic_uio.bdfs="2:0:4" | |
494 | ||
495 | This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in | |
496 | the next step. | |
497 | ||
498 | .. note:: | |
499 | ||
500 | Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs. | |
501 | ||
502 | #. Load nic_uio kernel driver: | |
503 | ||
504 | .. code-block:: console | |
505 | ||
506 | kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko | |
507 | ||
508 | #. Start testpmd with basic parameters: | |
509 | ||
510 | .. code-block:: console | |
511 | ||
512 | ./x86_64-native-bsdapp-clang/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i | |
513 | ||
514 | Example output: | |
515 | ||
516 | .. code-block:: console | |
517 | ||
518 | [...] | |
519 | EAL: PCI device 0000:02:00.4 on NUMA socket 0 | |
520 | EAL: probe driver: 1425:5401 rte_cxgbe_pmd | |
521 | EAL: PCI memory mapped at 0x8007ec000 | |
522 | EAL: PCI memory mapped at 0x842800000 | |
523 | EAL: PCI memory mapped at 0x80086c000 | |
524 | PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8 | |
525 | PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter | |
526 | Interactive-mode selected | |
527 | Configuring Port 0 (socket 0) | |
528 | Port 0: 00:07:43:2D:EA:C0 | |
529 | Configuring Port 1 (socket 0) | |
530 | Port 1: 00:07:43:2D:EA:C8 | |
531 | Checking link statuses... | |
532 | PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted | |
533 | PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted | |
534 | Port 0 Link Up - speed 10000 Mbps - full-duplex | |
535 | Port 1 Link Up - speed 10000 Mbps - full-duplex | |
536 | Done | |
537 | testpmd> | |
538 | ||
539 | .. note:: | |
540 | ||
541 | Flow control pause TX/RX is disabled by default and can be enabled via | |
542 | testpmd. Refer section :ref:`flow-control` for more details. | |
543 | ||
544 | Sample Application Notes | |
545 | ------------------------ | |
546 | ||
547 | .. _flow-control: | |
548 | ||
549 | Enable/Disable Flow Control | |
550 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
551 | ||
552 | Flow control pause TX/RX is disabled by default and can be enabled via | |
553 | testpmd as follows: | |
554 | ||
555 | .. code-block:: console | |
556 | ||
557 | testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0 | |
558 | testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1 | |
559 | ||
560 | To disable again, run: | |
561 | ||
562 | .. code-block:: console | |
563 | ||
564 | testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0 | |
565 | testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1 | |
566 | ||
567 | Jumbo Mode | |
568 | ~~~~~~~~~~ | |
569 | ||
570 | There are two ways to enable sending and receiving of jumbo frames via testpmd. | |
571 | One method involves using the **mtu** command, which changes the mtu of an | |
572 | individual port without having to stop the selected port. Another method | |
573 | involves stopping all the ports first and then running **max-pkt-len** command | |
574 | to configure the mtu of all the ports with a single command. | |
575 | ||
576 | - To configure each port individually, run the mtu command as follows: | |
577 | ||
578 | .. code-block:: console | |
579 | ||
580 | testpmd> port config mtu 0 9000 | |
581 | testpmd> port config mtu 1 9000 | |
582 | ||
583 | - To configure all the ports at once, stop all the ports first and run the | |
584 | max-pkt-len command as follows: | |
585 | ||
586 | .. code-block:: console | |
587 | ||
588 | testpmd> port stop all | |
589 | testpmd> port config all max-pkt-len 9000 |