ceph/src/spdk/dpdk/doc/guides/nics/cxgbe.rst

   1 .. SPDX-License-Identifier: BSD-3-Clause
   2    Copyright(c) 2014-2018 Chelsio Communications.
   3    All rights reserved.
   4
   5 CXGBE Poll Mode Driver
   6 ======================
   7
   8 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
   9 for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD
  10 has support for the latest Linux and FreeBSD operating systems.
  11
  12 CXGBEVF PMD provides poll mode driver support for SR-IOV Virtual functions
  13 and has support for the latest Linux operating systems.
  14
  15 More information can be found at `Chelsio Communications Official Website
  16 <http://www.chelsio.com>`_.
  17
  18 Features
  19 --------
  20
  21 CXGBE and CXGBEVF PMD has support for:
  22
  23 - Multiple queues for TX and RX
  24 - Receiver Side Steering (RSS)
  25   Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP.
  26   For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported.
  27 - VLAN filtering
  28 - Checksum offload
  29 - Promiscuous mode
  30 - All multicast mode
  31 - Port hardware statistics
  32 - Jumbo frames
  33 - Flow API - Support for both Wildcard (LE-TCAM) and Exact (HASH) match filters.
  34
  35 Limitations
  36 -----------
  37
  38 The Chelsio Terminator series of devices provide two/four ports but
  39 expose a single PCI bus address, thus, librte_pmd_cxgbe registers
  40 itself as a PCI driver that allocates one Ethernet device per detected
  41 port.
  42
  43 For this reason, one cannot whitelist/blacklist a single port without
  44 whitelisting/blacklisting the other ports on the same device.
  45
  46 .. _t5-nics:
  47
  48 Supported Chelsio T5 NICs
  49 -------------------------
  50
  51 - 1G NICs: T502-BT
  52 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
  53 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
  54 - Other T5 NICs: T522-CR
  55
  56 .. _t6-nics:
  57
  58 Supported Chelsio T6 NICs
  59 -------------------------
  60
  61 - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR
  62 - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR
  63
  64 Supported SR-IOV Chelsio NICs
  65 -----------------------------
  66
  67 SR-IOV virtual functions are supported on all the Chelsio NICs listed
  68 in :ref:`t5-nics` and :ref:`t6-nics`.
  69
  70 Prerequisites
  71 -------------
  72
  73 - Requires firmware version **1.17.14.0** and higher. Visit
  74   `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
  75   bundled with the latest Chelsio Unified Wire package.
  76
  77   For Linux, installing and loading the latest cxgb4 kernel driver from the
  78   Chelsio Unified Wire package should get you the latest firmware. More
  79   information can be obtained from the User Guide that is bundled with the
  80   Chelsio Unified Wire package.
  81
  82   For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
  83   package must be manually flashed via cxgbetool available in FreeBSD source
  84   repository.
  85
  86   Instructions on how to manually flash the firmware are given in section
  87   :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
  88   for FreeBSD.
  89
  90 Pre-Installation Configuration
  91 ------------------------------
  92
  93 Config File Options
  94 ~~~~~~~~~~~~~~~~~~~
  95
  96 The following options can be modified in the ``.config`` file. Please note that
  97 enabling debugging options may affect system performance.
  98
  99 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
 100
 101   Toggle compilation of librte_pmd_cxgbe driver.
 102
 103   .. note::
 104
 105      This controls compilation of both CXGBE and CXGBEVF PMD.
 106
 107 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
 108
 109   Toggle display of generic debugging messages.
 110
 111 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
 112
 113   Toggle display of registers related run-time check messages.
 114
 115 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
 116
 117   Toggle display of firmware mailbox related run-time check messages.
 118
 119 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
 120
 121   Toggle display of transmission data path run-time check messages.
 122
 123 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
 124
 125   Toggle display of receiving data path run-time check messages.
 126
 127 - ``CONFIG_RTE_LIBRTE_CXGBE_TPUT`` (default **y**)
 128
 129   Toggle behaviour to prefer Throughput or Latency.
 130
 131 Runtime Options
 132 ~~~~~~~~~~~~~~~
 133
 134 The following ``devargs`` options can be enabled at runtime. They must
 135 be passed as part of EAL arguments. For example,
 136
 137 .. code-block:: console
 138
 139    testpmd -w 02:00.4,keep_ovlan=1 -- -i
 140
 141 - ``keep_ovlan`` (default **0**)
 142
 143   Toggle behaviour to keep/strip outer VLAN in Q-in-Q packets. If
 144   enabled, the outer VLAN tag is preserved in Q-in-Q packets. Otherwise,
 145   the outer VLAN tag is stripped in Q-in-Q packets.
 146
 147 - ``force_link_up`` (default **0**)
 148
 149   When set to 1, CXGBEVF PMD always forces link as up for all VFs on
 150   underlying Chelsio NICs. This enables multiple VFs on the same NIC
 151   to send traffic to each other even when the physical link is down.
 152
 153 .. _driver-compilation:
 154
 155 Driver compilation and testing
 156 ------------------------------
 157
 158 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
 159 for details.
 160
 161 Linux
 162 -----
 163
 164 .. _linux-installation:
 165
 166 Linux Installation
 167 ~~~~~~~~~~~~~~~~~~
 168
 169 Steps to manually install the latest firmware from the downloaded Chelsio
 170 Unified Wire package for Linux operating system are as follows:
 171
 172 #. Load the kernel module:
 173
 174    .. code-block:: console
 175
 176       modprobe cxgb4
 177
 178 #. Use ifconfig to get the interface name assigned to Chelsio card:
 179
 180    .. code-block:: console
 181
 182       ifconfig -a | grep "00:07:43"
 183
 184    Example output:
 185
 186    .. code-block:: console
 187
 188       p1p1      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C0
 189       p1p2      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C8
 190
 191 #. Install cxgbtool:
 192
 193    .. code-block:: console
 194
 195       cd <path_to_uwire>/tools/cxgbtool
 196       make install
 197
 198 #. Use cxgbtool to load the firmware config file onto the card:
 199
 200    .. code-block:: console
 201
 202       cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
 203
 204 #. Use cxgbtool to load the firmware image onto the card:
 205
 206    .. code-block:: console
 207
 208       cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
 209
 210 #. Unload and reload the kernel module:
 211
 212    .. code-block:: console
 213
 214       modprobe -r cxgb4
 215       modprobe cxgb4
 216
 217 #. Verify with ethtool:
 218
 219    .. code-block:: console
 220
 221       ethtool -i p1p1 | grep "firmware"
 222
 223    Example output:
 224
 225    .. code-block:: console
 226
 227       firmware-version: 1.17.14.0, TP 0.1.4.9
 228
 229 Running testpmd
 230 ~~~~~~~~~~~~~~~
 231
 232 This section demonstrates how to launch **testpmd** with Chelsio
 233 devices managed by librte_pmd_cxgbe in Linux operating system.
 234
 235 #. Load the kernel module:
 236
 237    .. code-block:: console
 238
 239       modprobe cxgb4
 240
 241 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
 242
 243    .. code-block:: console
 244
 245       dmesg | tail -2
 246
 247    Example output:
 248
 249    .. code-block:: console
 250
 251       cxgb4 0000:02:00.4 p1p1: renamed from eth0
 252       cxgb4 0000:02:00.4 p1p2: renamed from eth1
 253
 254    .. note::
 255
 256       Both the interfaces of a Chelsio 2-port adapter are bound to the
 257       same PCI bus address.
 258
 259 #. Unload the kernel module:
 260
 261    .. code-block:: console
 262
 263       modprobe -ar cxgb4 csiostor
 264
 265 #. Running testpmd
 266
 267    Follow instructions available in the document
 268    :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
 269    to run testpmd.
 270
 271    .. note::
 272
 273       Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
 274
 275    Example output:
 276
 277    .. code-block:: console
 278
 279       [...]
 280       EAL: PCI device 0000:02:00.4 on NUMA socket -1
 281       EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
 282       EAL:   PCI memory mapped at 0x7fd7c0200000
 283       EAL:   PCI memory mapped at 0x7fd77cdfd000
 284       EAL:   PCI memory mapped at 0x7fd7c10b7000
 285       PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
 286       PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
 287       Interactive-mode selected
 288       Configuring Port 0 (socket 0)
 289       Port 0: 00:07:43:2D:EA:C0
 290       Configuring Port 1 (socket 0)
 291       Port 1: 00:07:43:2D:EA:C8
 292       Checking link statuses...
 293       PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
 294       PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
 295       Port 0 Link Up - speed 10000 Mbps - full-duplex
 296       Port 1 Link Up - speed 10000 Mbps - full-duplex
 297       Done
 298       testpmd>
 299
 300    .. note::
 301
 302       Flow control pause TX/RX is disabled by default and can be enabled via
 303       testpmd. Refer section :ref:`flow-control` for more details.
 304
 305 Configuring SR-IOV Virtual Functions
 306 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 307
 308 This section demonstrates how to enable SR-IOV virtual functions
 309 on Chelsio NICs and demonstrates how to run testpmd with SR-IOV
 310 virtual functions.
 311
 312 #. Load the kernel module:
 313
 314    .. code-block:: console
 315
 316       modprobe cxgb4
 317
 318 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
 319
 320    .. code-block:: console
 321
 322       dmesg | tail -2
 323
 324    Example output:
 325
 326    .. code-block:: console
 327
 328       cxgb4 0000:02:00.4 p1p1: renamed from eth0
 329       cxgb4 0000:02:00.4 p1p2: renamed from eth1
 330
 331    .. note::
 332
 333       Both the interfaces of a Chelsio 2-port adapter are bound to the
 334       same PCI bus address.
 335
 336 #. Use ifconfig to get the interface name assigned to Chelsio card:
 337
 338    .. code-block:: console
 339
 340       ifconfig -a | grep "00:07:43"
 341
 342    Example output:
 343
 344    .. code-block:: console
 345
 346       p1p1      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C0
 347       p1p2      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C8
 348
 349 #. Bring up the interfaces:
 350
 351    .. code-block:: console
 352
 353       ifconfig p1p1 up
 354       ifconfig p1p2 up
 355
 356 #. Instantiate SR-IOV Virtual Functions. PF0..3 can be used for
 357    SR-IOV VFs. Multiple VFs can be instantiated on each of PF0..3.
 358    To instantiate one SR-IOV VF on each PF0 and PF1:
 359
 360    .. code-block:: console
 361
 362       echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs
 363       echo 1 > /sys/bus/pci/devices/0000\:02\:00.1/sriov_numvfs
 364
 365 #. Get the PCI bus addresses of the virtual functions:
 366
 367    .. code-block:: console
 368
 369       lspci | grep -i "Chelsio" | grep -i "VF"
 370
 371    Example output:
 372
 373    .. code-block:: console
 374
 375       02:01.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
 376       02:01.1 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF]
 377
 378 #. Running testpmd
 379
 380    Follow instructions available in the document
 381    :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
 382    to bind virtual functions and run testpmd.
 383
 384    Example output:
 385
 386    .. code-block:: console
 387
 388       [...]
 389       EAL: PCI device 0000:02:01.0 on NUMA socket 0
 390       EAL:   probe driver: 1425:5803 net_cxgbevf
 391       PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
 392       PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
 393       PMD: rte_cxgbe_pmd: Chelsio rev 0
 394       PMD: rte_cxgbe_pmd: No bootstrap loaded
 395       PMD: rte_cxgbe_pmd: No Expansion ROM loaded
 396       PMD: rte_cxgbe_pmd:  0000:02:01.0 Chelsio rev 0 1G/10GBASE-SFP
 397       EAL: PCI device 0000:02:01.1 on NUMA socket 0
 398       EAL:   probe driver: 1425:5803 net_cxgbevf
 399       PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0
 400       PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9
 401       PMD: rte_cxgbe_pmd: Chelsio rev 0
 402       PMD: rte_cxgbe_pmd: No bootstrap loaded
 403       PMD: rte_cxgbe_pmd: No Expansion ROM loaded
 404       PMD: rte_cxgbe_pmd:  0000:02:01.1 Chelsio rev 0 1G/10GBASE-SFP
 405       Configuring Port 0 (socket 0)
 406       Port 0: 06:44:29:44:40:00
 407       Configuring Port 1 (socket 0)
 408       Port 1: 06:44:29:44:40:10
 409       Checking link statuses...
 410       Done
 411       testpmd>
 412
 413 FreeBSD
 414 -------
 415
 416 .. _freebsd-installation:
 417
 418 FreeBSD Installation
 419 ~~~~~~~~~~~~~~~~~~~~
 420
 421 Steps to manually install the latest firmware from the downloaded Chelsio
 422 Unified Wire package for FreeBSD operating system are as follows:
 423
 424 #. Load the kernel module:
 425
 426    .. code-block:: console
 427
 428       kldload if_cxgbe
 429
 430 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
 431
 432    .. code-block:: console
 433
 434       dmesg | grep "t5nex"
 435
 436    Example output:
 437
 438    .. code-block:: console
 439
 440       t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
 441       cxl0: <port 0> on t5nex0
 442       cxl1: <port 1> on t5nex0
 443       t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
 444
 445    In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
 446
 447 #. Install cxgbetool from FreeBSD source repository:
 448
 449    .. code-block:: console
 450
 451       cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
 452       make && make install
 453
 454 #. Use cxgbetool to load the firmware image onto the card:
 455
 456    .. code-block:: console
 457
 458       cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
 459
 460 #. Unload and reload the kernel module:
 461
 462    .. code-block:: console
 463
 464       kldunload if_cxgbe
 465       kldload if_cxgbe
 466
 467 #. Verify with sysctl:
 468
 469    .. code-block:: console
 470
 471       sysctl -a | grep "t5nex" | grep "firmware"
 472
 473    Example output:
 474
 475    .. code-block:: console
 476
 477       dev.t5nex.0.firmware_version: 1.17.14.0
 478
 479 Running testpmd
 480 ~~~~~~~~~~~~~~~
 481
 482 This section demonstrates how to launch **testpmd** with Chelsio
 483 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
 484
 485 #. Change to DPDK source directory where the target has been compiled in
 486    section :ref:`driver-compilation`:
 487
 488    .. code-block:: console
 489
 490       cd <DPDK-source-directory>
 491
 492 #. Copy the contigmem kernel module to /boot/kernel directory:
 493
 494    .. code-block:: console
 495
 496       cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
 497
 498 #. Add the following lines to /boot/loader.conf:
 499
 500    .. code-block:: console
 501
 502       # reserve 2 x 1G blocks of contiguous memory using contigmem driver
 503       hw.contigmem.num_buffers=2
 504       hw.contigmem.buffer_size=1073741824
 505       # load contigmem module during boot process
 506       contigmem_load="YES"
 507
 508    The above lines load the contigmem kernel module during boot process and
 509    allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
 510    This is to avoid issues with potential memory fragmentation during later
 511    system up time, which may result in failure of allocating the contiguous
 512    memory required for the contigmem kernel module.
 513
 514 #. Restart the system and ensure the contigmem module is loaded successfully:
 515
 516    .. code-block:: console
 517
 518       reboot
 519       kldstat | grep "contigmem"
 520
 521    Example output:
 522
 523    .. code-block:: console
 524
 525       2    1 0xffffffff817f1000 3118     contigmem.ko
 526
 527 #. Repeat step 1 to ensure that you are in the DPDK source directory.
 528
 529 #. Load the cxgbe kernel module:
 530
 531    .. code-block:: console
 532
 533       kldload if_cxgbe
 534
 535 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
 536
 537    .. code-block:: console
 538
 539       pciconf -l | grep "t5nex"
 540
 541    Example output:
 542
 543    .. code-block:: console
 544
 545       t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
 546
 547    In the above example, the t5nex0 is bound to 2:0:4 bus address.
 548
 549    .. note::
 550
 551       Both the interfaces of a Chelsio 2-port adapter are bound to the
 552       same PCI bus address.
 553
 554 #. Unload the kernel module:
 555
 556    .. code-block:: console
 557
 558       kldunload if_cxgbe
 559
 560 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
 561
 562    .. code-block:: console
 563
 564       kenv hw.nic_uio.bdfs="2:0:4"
 565
 566    This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
 567    the next step.
 568
 569    .. note::
 570
 571       Currently, CXGBE PMD only supports the binding of PF4 for Chelsio NICs.
 572
 573 #. Load nic_uio kernel driver:
 574
 575    .. code-block:: console
 576
 577       kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
 578
 579 #. Start testpmd with basic parameters:
 580
 581    .. code-block:: console
 582
 583       ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
 584
 585    Example output:
 586
 587    .. code-block:: console
 588
 589       [...]
 590       EAL: PCI device 0000:02:00.4 on NUMA socket 0
 591       EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
 592       EAL:   PCI memory mapped at 0x8007ec000
 593       EAL:   PCI memory mapped at 0x842800000
 594       EAL:   PCI memory mapped at 0x80086c000
 595       PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9
 596       PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
 597       Interactive-mode selected
 598       Configuring Port 0 (socket 0)
 599       Port 0: 00:07:43:2D:EA:C0
 600       Configuring Port 1 (socket 0)
 601       Port 1: 00:07:43:2D:EA:C8
 602       Checking link statuses...
 603       PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
 604       PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
 605       Port 0 Link Up - speed 10000 Mbps - full-duplex
 606       Port 1 Link Up - speed 10000 Mbps - full-duplex
 607       Done
 608       testpmd>
 609
 610 .. note::
 611
 612    Flow control pause TX/RX is disabled by default and can be enabled via
 613    testpmd. Refer section :ref:`flow-control` for more details.
 614
 615 Sample Application Notes
 616 ------------------------
 617
 618 .. _flow-control:
 619
 620 Enable/Disable Flow Control
 621 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 622
 623 Flow control pause TX/RX is disabled by default and can be enabled via
 624 testpmd as follows:
 625
 626 .. code-block:: console
 627
 628    testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
 629    testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
 630
 631 To disable again, run:
 632
 633 .. code-block:: console
 634
 635    testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
 636    testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
 637
 638 Jumbo Mode
 639 ~~~~~~~~~~
 640
 641 There are two ways to enable sending and receiving of jumbo frames via testpmd.
 642 One method involves using the **mtu** command, which changes the mtu of an
 643 individual port without having to stop the selected port. Another method
 644 involves stopping all the ports first and then running **max-pkt-len** command
 645 to configure the mtu of all the ports with a single command.
 646
 647 - To configure each port individually, run the mtu command as follows:
 648
 649   .. code-block:: console
 650
 651      testpmd> port config mtu 0 9000
 652      testpmd> port config mtu 1 9000
 653
 654 - To configure all the ports at once, stop all the ports first and run the
 655   max-pkt-len command as follows:
 656
 657   .. code-block:: console
 658
 659      testpmd> port stop all
 660      testpmd> port config all max-pkt-len 9000