ceph/src/seastar/dpdk/doc/guides/nics/cxgbe.rst

   1 ..  BSD LICENSE
   2     Copyright 2015 Chelsio Communications.
   3     All rights reserved.
   4
   5     Redistribution and use in source and binary forms, with or without
   6     modification, are permitted provided that the following conditions
   7     are met:
   8
   9     * Redistributions of source code must retain the above copyright
  10     notice, this list of conditions and the following disclaimer.
  11     * Redistributions in binary form must reproduce the above copyright
  12     notice, this list of conditions and the following disclaimer in
  13     the documentation and/or other materials provided with the
  14     distribution.
  15     * Neither the name of Chelsio Communications nor the names of its
  16     contributors may be used to endorse or promote products derived
  17     from this software without specific prior written permission.
  18
  19     THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
  20     "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
  21     LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
  22     A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
  23     OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  24     SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
  25     LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
  26     DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
  27     THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  28     (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
  29     OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  30
  31 CXGBE Poll Mode Driver
  32 ======================
  33
  34 The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
  35 for **Chelsio T5** 10/40 Gbps family of adapters. CXGBE PMD has support
  36 for the latest Linux and FreeBSD operating systems.
  37
  38 More information can be found at `Chelsio Communications Official Website
  39 <http://www.chelsio.com>`_.
  40
  41 Features
  42 --------
  43
  44 CXGBE PMD has support for:
  45
  46 - Multiple queues for TX and RX
  47 - Receiver Side Steering (RSS)
  48 - VLAN filtering
  49 - Checksum offload
  50 - Promiscuous mode
  51 - All multicast mode
  52 - Port hardware statistics
  53 - Jumbo frames
  54
  55 Limitations
  56 -----------
  57
  58 The Chelsio T5 devices provide two/four ports but expose a single PCI bus
  59 address, thus, librte_pmd_cxgbe registers itself as a
  60 PCI driver that allocates one Ethernet device per detected port.
  61
  62 For this reason, one cannot whitelist/blacklist a single port without
  63 whitelisting/blacklisting the other ports on the same device.
  64
  65 Supported Chelsio T5 NICs
  66 -------------------------
  67
  68 - 1G NICs: T502-BT
  69 - 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
  70 - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
  71 - Other T5 NICs: T522-CR
  72
  73 Prerequisites
  74 -------------
  75
  76 - Requires firmware version **1.13.32.0** and higher. Visit
  77   `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
  78   bundled with the latest Chelsio Unified Wire package.
  79
  80   For Linux, installing and loading the latest cxgb4 kernel driver from the
  81   Chelsio Unified Wire package should get you the latest firmware. More
  82   information can be obtained from the User Guide that is bundled with the
  83   Chelsio Unified Wire package.
  84
  85   For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
  86   package must be manually flashed via cxgbetool available in FreeBSD source
  87   repository.
  88
  89   Instructions on how to manually flash the firmware are given in section
  90   :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
  91   for FreeBSD.
  92
  93 Pre-Installation Configuration
  94 ------------------------------
  95
  96 Config File Options
  97 ~~~~~~~~~~~~~~~~~~~
  98
  99 The following options can be modified in the ``.config`` file. Please note that
 100 enabling debugging options may affect system performance.
 101
 102 - ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
 103
 104   Toggle compilation of librte_pmd_cxgbe driver.
 105
 106 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
 107
 108   Toggle display of generic debugging messages.
 109
 110 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
 111
 112   Toggle display of registers related run-time check messages.
 113
 114 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
 115
 116   Toggle display of firmware mailbox related run-time check messages.
 117
 118 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
 119
 120   Toggle display of transmission data path run-time check messages.
 121
 122 - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
 123
 124   Toggle display of receiving data path run-time check messages.
 125
 126 .. _driver-compilation:
 127
 128 Driver compilation and testing
 129 ------------------------------
 130
 131 Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
 132 for details.
 133
 134 Linux
 135 -----
 136
 137 .. _linux-installation:
 138
 139 Linux Installation
 140 ~~~~~~~~~~~~~~~~~~
 141
 142 Steps to manually install the latest firmware from the downloaded Chelsio
 143 Unified Wire package for Linux operating system are as follows:
 144
 145 #. Load the kernel module:
 146
 147    .. code-block:: console
 148
 149       modprobe cxgb4
 150
 151 #. Use ifconfig to get the interface name assigned to Chelsio card:
 152
 153    .. code-block:: console
 154
 155       ifconfig -a | grep "00:07:43"
 156
 157    Example output:
 158
 159    .. code-block:: console
 160
 161       p1p1      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C0
 162       p1p2      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C8
 163
 164 #. Install cxgbtool:
 165
 166    .. code-block:: console
 167
 168       cd <path_to_uwire>/tools/cxgbtool
 169       make install
 170
 171 #. Use cxgbtool to load the firmware config file onto the card:
 172
 173    .. code-block:: console
 174
 175       cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
 176
 177 #. Use cxgbtool to load the firmware image onto the card:
 178
 179    .. code-block:: console
 180
 181       cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
 182
 183 #. Unload and reload the kernel module:
 184
 185    .. code-block:: console
 186
 187       modprobe -r cxgb4
 188       modprobe cxgb4
 189
 190 #. Verify with ethtool:
 191
 192    .. code-block:: console
 193
 194       ethtool -i p1p1 | grep "firmware"
 195
 196    Example output:
 197
 198    .. code-block:: console
 199
 200       firmware-version: 1.13.32.0, TP 0.1.4.8
 201
 202 Running testpmd
 203 ~~~~~~~~~~~~~~~
 204
 205 This section demonstrates how to launch **testpmd** with Chelsio T5
 206 devices managed by librte_pmd_cxgbe in Linux operating system.
 207
 208 #. Load the kernel module:
 209
 210    .. code-block:: console
 211
 212       modprobe cxgb4
 213
 214 #. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
 215
 216    .. code-block:: console
 217
 218       dmesg | tail -2
 219
 220    Example output:
 221
 222    .. code-block:: console
 223
 224       cxgb4 0000:02:00.4 p1p1: renamed from eth0
 225       cxgb4 0000:02:00.4 p1p2: renamed from eth1
 226
 227    .. note::
 228
 229       Both the interfaces of a Chelsio T5 2-port adapter are bound to the
 230       same PCI bus address.
 231
 232 #. Unload the kernel module:
 233
 234    .. code-block:: console
 235
 236       modprobe -ar cxgb4 csiostor
 237
 238 #. Running testpmd
 239
 240    Follow instructions available in the document
 241    :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
 242    to run testpmd.
 243
 244    .. note::
 245
 246       Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
 247
 248    Example output:
 249
 250    .. code-block:: console
 251
 252       [...]
 253       EAL: PCI device 0000:02:00.4 on NUMA socket -1
 254       EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
 255       EAL:   PCI memory mapped at 0x7fd7c0200000
 256       EAL:   PCI memory mapped at 0x7fd77cdfd000
 257       EAL:   PCI memory mapped at 0x7fd7c10b7000
 258       PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8
 259       PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
 260       Interactive-mode selected
 261       Configuring Port 0 (socket 0)
 262       Port 0: 00:07:43:2D:EA:C0
 263       Configuring Port 1 (socket 0)
 264       Port 1: 00:07:43:2D:EA:C8
 265       Checking link statuses...
 266       PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
 267       PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
 268       Port 0 Link Up - speed 10000 Mbps - full-duplex
 269       Port 1 Link Up - speed 10000 Mbps - full-duplex
 270       Done
 271       testpmd>
 272
 273    .. note::
 274
 275       Flow control pause TX/RX is disabled by default and can be enabled via
 276       testpmd. Refer section :ref:`flow-control` for more details.
 277
 278 FreeBSD
 279 -------
 280
 281 .. _freebsd-installation:
 282
 283 FreeBSD Installation
 284 ~~~~~~~~~~~~~~~~~~~~
 285
 286 Steps to manually install the latest firmware from the downloaded Chelsio
 287 Unified Wire package for FreeBSD operating system are as follows:
 288
 289 #. Load the kernel module:
 290
 291    .. code-block:: console
 292
 293       kldload if_cxgbe
 294
 295 #. Use dmesg to get the t5nex instance assigned to the Chelsio card:
 296
 297    .. code-block:: console
 298
 299       dmesg | grep "t5nex"
 300
 301    Example output:
 302
 303    .. code-block:: console
 304
 305       t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
 306       cxl0: <port 0> on t5nex0
 307       cxl1: <port 1> on t5nex0
 308       t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
 309
 310    In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
 311
 312 #. Install cxgbetool from FreeBSD source repository:
 313
 314    .. code-block:: console
 315
 316       cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
 317       make && make install
 318
 319 #. Use cxgbetool to load the firmware image onto the card:
 320
 321    .. code-block:: console
 322
 323       cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
 324
 325 #. Unload and reload the kernel module:
 326
 327    .. code-block:: console
 328
 329       kldunload if_cxgbe
 330       kldload if_cxgbe
 331
 332 #. Verify with sysctl:
 333
 334    .. code-block:: console
 335
 336       sysctl -a | grep "t5nex" | grep "firmware"
 337
 338    Example output:
 339
 340    .. code-block:: console
 341
 342       dev.t5nex.0.firmware_version: 1.13.32.0
 343
 344 Running testpmd
 345 ~~~~~~~~~~~~~~~
 346
 347 This section demonstrates how to launch **testpmd** with Chelsio T5
 348 devices managed by librte_pmd_cxgbe in FreeBSD operating system.
 349
 350 #. Change to DPDK source directory where the target has been compiled in
 351    section :ref:`driver-compilation`:
 352
 353    .. code-block:: console
 354
 355       cd <DPDK-source-directory>
 356
 357 #. Copy the contigmem kernel module to /boot/kernel directory:
 358
 359    .. code-block:: console
 360
 361       cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
 362
 363 #. Add the following lines to /boot/loader.conf:
 364
 365    .. code-block:: console
 366
 367       # reserve 2 x 1G blocks of contiguous memory using contigmem driver
 368       hw.contigmem.num_buffers=2
 369       hw.contigmem.buffer_size=1073741824
 370       # load contigmem module during boot process
 371       contigmem_load="YES"
 372
 373    The above lines load the contigmem kernel module during boot process and
 374    allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
 375    This is to avoid issues with potential memory fragmentation during later
 376    system up time, which may result in failure of allocating the contiguous
 377    memory required for the contigmem kernel module.
 378
 379 #. Restart the system and ensure the contigmem module is loaded successfully:
 380
 381    .. code-block:: console
 382
 383       reboot
 384       kldstat | grep "contigmem"
 385
 386    Example output:
 387
 388    .. code-block:: console
 389
 390       2    1 0xffffffff817f1000 3118     contigmem.ko
 391
 392 #. Repeat step 1 to ensure that you are in the DPDK source directory.
 393
 394 #. Load the cxgbe kernel module:
 395
 396    .. code-block:: console
 397
 398       kldload if_cxgbe
 399
 400 #. Get the PCI bus addresses of the interfaces bound to t5nex driver:
 401
 402    .. code-block:: console
 403
 404       pciconf -l | grep "t5nex"
 405
 406    Example output:
 407
 408    .. code-block:: console
 409
 410       t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
 411
 412    In the above example, the t5nex0 is bound to 2:0:4 bus address.
 413
 414    .. note::
 415
 416       Both the interfaces of a Chelsio T5 2-port adapter are bound to the
 417       same PCI bus address.
 418
 419 #. Unload the kernel module:
 420
 421    .. code-block:: console
 422
 423       kldunload if_cxgbe
 424
 425 #. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
 426
 427    .. code-block:: console
 428
 429       kenv hw.nic_uio.bdfs="2:0:4"
 430
 431    This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
 432    the next step.
 433
 434    .. note::
 435
 436       Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
 437
 438 #. Load nic_uio kernel driver:
 439
 440    .. code-block:: console
 441
 442       kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
 443
 444 #. Start testpmd with basic parameters:
 445
 446    .. code-block:: console
 447
 448       ./x86_64-native-bsdapp-clang/app/testpmd -l 0-3 -n 4 -w 0000:02:00.4 -- -i
 449
 450    Example output:
 451
 452    .. code-block:: console
 453
 454       [...]
 455       EAL: PCI device 0000:02:00.4 on NUMA socket 0
 456       EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
 457       EAL:   PCI memory mapped at 0x8007ec000
 458       EAL:   PCI memory mapped at 0x842800000
 459       EAL:   PCI memory mapped at 0x80086c000
 460       PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8
 461       PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
 462       Interactive-mode selected
 463       Configuring Port 0 (socket 0)
 464       Port 0: 00:07:43:2D:EA:C0
 465       Configuring Port 1 (socket 0)
 466       Port 1: 00:07:43:2D:EA:C8
 467       Checking link statuses...
 468       PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
 469       PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
 470       Port 0 Link Up - speed 10000 Mbps - full-duplex
 471       Port 1 Link Up - speed 10000 Mbps - full-duplex
 472       Done
 473       testpmd>
 474
 475 .. note::
 476
 477    Flow control pause TX/RX is disabled by default and can be enabled via
 478    testpmd. Refer section :ref:`flow-control` for more details.
 479
 480 Sample Application Notes
 481 ------------------------
 482
 483 .. _flow-control:
 484
 485 Enable/Disable Flow Control
 486 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 487
 488 Flow control pause TX/RX is disabled by default and can be enabled via
 489 testpmd as follows:
 490
 491 .. code-block:: console
 492
 493    testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
 494    testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
 495
 496 To disable again, run:
 497
 498 .. code-block:: console
 499
 500    testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
 501    testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
 502
 503 Jumbo Mode
 504 ~~~~~~~~~~
 505
 506 There are two ways to enable sending and receiving of jumbo frames via testpmd.
 507 One method involves using the **mtu** command, which changes the mtu of an
 508 individual port without having to stop the selected port. Another method
 509 involves stopping all the ports first and then running **max-pkt-len** command
 510 to configure the mtu of all the ports with a single command.
 511
 512 - To configure each port individually, run the mtu command as follows:
 513
 514   .. code-block:: console
 515
 516      testpmd> port config mtu 0 9000
 517      testpmd> port config mtu 1 9000
 518
 519 - To configure all the ports at once, stop all the ports first and run the
 520   max-pkt-len command as follows:
 521
 522   .. code-block:: console
 523
 524      testpmd> port stop all
 525      testpmd> port config all max-pkt-len 9000