ceph/src/spdk/dpdk/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst

   1 ..  SPDX-License-Identifier: BSD-3-Clause
   2     Copyright(c) 2010-2014 Intel Corporation.
   3
   4 VMDQ and DCB Forwarding Sample Application
   5 ==========================================
   6
   7 The VMDQ and DCB Forwarding sample application is a simple example of packet processing using the DPDK.
   8 The application performs L2 forwarding using VMDQ and DCB to divide the incoming traffic into queues.
   9 The traffic splitting is performed in hardware by the VMDQ and DCB features of the Intel® 82599 and X710/XL710 Ethernet Controllers.
  10
  11 Overview
  12 --------
  13
  14 This sample application can be used as a starting point for developing a new application that is based on the DPDK and
  15 uses VMDQ and DCB for traffic partitioning.
  16
  17 The VMDQ and DCB filters work on MAC and VLAN traffic to divide the traffic into input queues on the basis of the Destination MAC
  18 address, VLAN ID and VLAN user priority fields.
  19 VMDQ filters split the traffic into 16 or 32 groups based on the Destination MAC and VLAN ID.
  20 Then, DCB places each packet into one of queues within that group, based upon the VLAN user priority field.
  21
  22 All traffic is read from a single incoming port (port 0) and output on port 1, without any processing being performed.
  23 With Intel® 82599 NIC, for example, the traffic is split into 128 queues on input, where each thread of the application reads from
  24 multiple queues. When run with 8 threads, that is, with the -c FF option, each thread receives and forwards packets from 16 queues.
  25
  26 As supplied, the sample application configures the VMDQ feature to have 32 pools with 4 queues each as indicated in :numref:`figure_vmdq_dcb_example`.
  27 The Intel® 82599 10 Gigabit Ethernet Controller NIC also supports the splitting of traffic into 16 pools of 8 queues. While the
  28 Intel® X710 or XL710 Ethernet Controller NICs support many configurations of VMDQ pools of 4 or 8 queues each. For simplicity, only 16
  29 or 32 pools is supported in this sample. And queues numbers for each VMDQ pool can be changed by setting CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM
  30 in config/common_* file.
  31 The nb-pools, nb-tcs and enable-rss parameters can be passed on the command line, after the EAL parameters:
  32
  33 .. code-block:: console
  34
  35     ./build/vmdq_dcb [EAL options] -- -p PORTMASK --nb-pools NP --nb-tcs TC --enable-rss
  36
  37 where, NP can be 16 or 32, TC can be 4 or 8, rss is disabled by default.
  38
  39 .. _figure_vmdq_dcb_example:
  40
  41 .. figure:: img/vmdq_dcb_example.*
  42
  43    Packet Flow Through the VMDQ and DCB Sample Application
  44
  45
  46 In Linux* user space, the application can display statistics with the number of packets received on each queue.
  47 To have the application display the statistics, send a SIGHUP signal to the running application process.
  48
  49 The VMDQ and DCB Forwarding sample application is in many ways simpler than the L2 Forwarding application
  50 (see :doc:`l2_forward_real_virtual`)
  51 as it performs unidirectional L2 forwarding of packets from one port to a second port.
  52 No command-line options are taken by this application apart from the standard EAL command-line options.
  53
  54 .. note::
  55
  56     Since VMD queues are being used for VMM, this application works correctly
  57     when VTd is disabled in the BIOS or Linux* kernel (intel_iommu=off).
  58
  59 Compiling the Application
  60 -------------------------
  61
  62
  63
  64 To compile the sample application see :doc:`compiling`.
  65
  66 The application is located in the ``vmdq_dcb`` sub-directory.
  67
  68 Running the Application
  69 -----------------------
  70
  71 To run the example in a linux environment:
  72
  73 .. code-block:: console
  74
  75     user@target:~$ ./build/vmdq_dcb -l 0-3 -n 4 -- -p 0x3 --nb-pools 32 --nb-tcs 4
  76
  77 Refer to the *DPDK Getting Started Guide* for general information on running applications and
  78 the Environment Abstraction Layer (EAL) options.
  79
  80 Explanation
  81 -----------
  82
  83 The following sections provide some explanation of the code.
  84
  85 Initialization
  86 ~~~~~~~~~~~~~~
  87
  88 The EAL, driver and PCI configuration is performed largely as in the L2 Forwarding sample application,
  89 as is the creation of the mbuf pool.
  90 See :doc:`l2_forward_real_virtual`.
  91 Where this example application differs is in the configuration of the NIC port for RX.
  92
  93 The VMDQ and DCB hardware feature is configured at port initialization time by setting the appropriate values in the
  94 rte_eth_conf structure passed to the rte_eth_dev_configure() API.
  95 Initially in the application,
  96 a default structure is provided for VMDQ and DCB configuration to be filled in later by the application.
  97
  98 .. code-block:: c
  99
 100     /* empty vmdq+dcb configuration structure. Filled in programmatically */
 101     static const struct rte_eth_conf vmdq_dcb_conf_default = {
 102         .rxmode = {
 103             .mq_mode        = ETH_MQ_RX_VMDQ_DCB,
 104             .split_hdr_size = 0,
 105         },
 106         .txmode = {
 107             .mq_mode = ETH_MQ_TX_VMDQ_DCB,
 108         },
 109         /*
 110          * should be overridden separately in code with
 111          * appropriate values
 112          */
 113         .rx_adv_conf = {
 114             .vmdq_dcb_conf = {
 115                 .nb_queue_pools = ETH_32_POOLS,
 116                 .enable_default_pool = 0,
 117                 .default_pool = 0,
 118                 .nb_pool_maps = 0,
 119                 .pool_map = {{0, 0},},
 120                 .dcb_tc = {0},
 121             },
 122             .dcb_rx_conf = {
 123                 .nb_tcs = ETH_4_TCS,
 124                 /** Traffic class each UP mapped to. */
 125                 .dcb_tc = {0},
 126             },
 127             .vmdq_rx_conf = {
 128                 .nb_queue_pools = ETH_32_POOLS,
 129                 .enable_default_pool = 0,
 130                 .default_pool = 0,
 131                 .nb_pool_maps = 0,
 132                 .pool_map = {{0, 0},},
 133             },
 134         },
 135         .tx_adv_conf = {
 136             .vmdq_dcb_tx_conf = {
 137                 .nb_queue_pools = ETH_32_POOLS,
 138                 .dcb_tc = {0},
 139             },
 140         },
 141     };
 142
 143 The get_eth_conf() function fills in an rte_eth_conf structure with the appropriate values,
 144 based on the global vlan_tags array,
 145 and dividing up the possible user priority values equally among the individual queues
 146 (also referred to as traffic classes) within each pool. With Intel® 82599 NIC,
 147 if the number of pools is 32, then the user priority fields are allocated 2 to a queue.
 148 If 16 pools are used, then each of the 8 user priority fields is allocated to its own queue within the pool.
 149 With Intel® X710/XL710 NICs, if number of tcs is 4, and number of queues in pool is 8,
 150 then the user priority fields are allocated 2 to one tc, and a tc has 2 queues mapping to it, then
 151 RSS will determine the destination queue in 2.
 152 For the VLAN IDs, each one can be allocated to possibly multiple pools of queues,
 153 so the pools parameter in the rte_eth_vmdq_dcb_conf structure is specified as a bitmask value.
 154 For destination MAC, each VMDQ pool will be assigned with a MAC address. In this sample, each VMDQ pool
 155 is assigned to the MAC like 52:54:00:12:<port_id>:<pool_id>, that is,
 156 the MAC of VMDQ pool 2 on port 1 is 52:54:00:12:01:02.
 157
 158 .. code-block:: c
 159
 160     const uint16_t vlan_tags[] = {
 161         0, 1, 2, 3, 4, 5, 6, 7,
 162         8, 9, 10, 11, 12, 13, 14, 15,
 163         16, 17, 18, 19, 20, 21, 22, 23,
 164         24, 25, 26, 27, 28, 29, 30, 31
 165     };
 166
 167     /* pool mac addr template, pool mac addr is like: 52 54 00 12 port# pool# */
 168     static struct rte_ether_addr pool_addr_template = {
 169         .addr_bytes = {0x52, 0x54, 0x00, 0x12, 0x00, 0x00}
 170     };
 171
 172     /* Builds up the correct configuration for vmdq+dcb based on the vlan tags array
 173      * given above, and the number of traffic classes available for use. */
 174     static inline int
 175     get_eth_conf(struct rte_eth_conf *eth_conf)
 176     {
 177         struct rte_eth_vmdq_dcb_conf conf;
 178         struct rte_eth_vmdq_rx_conf  vmdq_conf;
 179         struct rte_eth_dcb_rx_conf   dcb_conf;
 180         struct rte_eth_vmdq_dcb_tx_conf tx_conf;
 181         uint8_t i;
 182
 183         conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools;
 184         vmdq_conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools;
 185         tx_conf.nb_queue_pools = (enum rte_eth_nb_pools)num_pools;
 186         conf.nb_pool_maps = num_pools;
 187         vmdq_conf.nb_pool_maps = num_pools;
 188         conf.enable_default_pool = 0;
 189         vmdq_conf.enable_default_pool = 0;
 190         conf.default_pool = 0; /* set explicit value, even if not used */
 191         vmdq_conf.default_pool = 0;
 192
 193         for (i = 0; i < conf.nb_pool_maps; i++) {
 194             conf.pool_map[i].vlan_id = vlan_tags[i];
 195             vmdq_conf.pool_map[i].vlan_id = vlan_tags[i];
 196             conf.pool_map[i].pools = 1UL << i ;
 197             vmdq_conf.pool_map[i].pools = 1UL << i;
 198         }
 199         for (i = 0; i < ETH_DCB_NUM_USER_PRIORITIES; i++){
 200             conf.dcb_tc[i] = i % num_tcs;
 201             dcb_conf.dcb_tc[i] = i % num_tcs;
 202             tx_conf.dcb_tc[i] = i % num_tcs;
 203         }
 204         dcb_conf.nb_tcs = (enum rte_eth_nb_tcs)num_tcs;
 205         (void)(rte_memcpy(eth_conf, &vmdq_dcb_conf_default, sizeof(*eth_conf)));
 206         (void)(rte_memcpy(&eth_conf->rx_adv_conf.vmdq_dcb_conf, &conf,
 207                   sizeof(conf)));
 208         (void)(rte_memcpy(&eth_conf->rx_adv_conf.dcb_rx_conf, &dcb_conf,
 209                   sizeof(dcb_conf)));
 210         (void)(rte_memcpy(&eth_conf->rx_adv_conf.vmdq_rx_conf, &vmdq_conf,
 211                   sizeof(vmdq_conf)));
 212         (void)(rte_memcpy(&eth_conf->tx_adv_conf.vmdq_dcb_tx_conf, &tx_conf,
 213                   sizeof(tx_conf)));
 214         if (rss_enable) {
 215             eth_conf->rxmode.mq_mode= ETH_MQ_RX_VMDQ_DCB_RSS;
 216             eth_conf->rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP |
 217                                 ETH_RSS_UDP |
 218                                 ETH_RSS_TCP |
 219                                 ETH_RSS_SCTP;
 220         }
 221         return 0;
 222     }
 223
 224     ......
 225
 226     /* Set mac for each pool.*/
 227     for (q = 0; q < num_pools; q++) {
 228         struct rte_ether_addr mac;
 229         mac = pool_addr_template;
 230         mac.addr_bytes[4] = port;
 231         mac.addr_bytes[5] = q;
 232         printf("Port %u vmdq pool %u set mac %02x:%02x:%02x:%02x:%02x:%02x\n",
 233             port, q,
 234             mac.addr_bytes[0], mac.addr_bytes[1],
 235             mac.addr_bytes[2], mac.addr_bytes[3],
 236             mac.addr_bytes[4], mac.addr_bytes[5]);
 237         retval = rte_eth_dev_mac_addr_add(port, &mac,
 238                 q + vmdq_pool_base);
 239         if (retval) {
 240             printf("mac addr add failed at pool %d\n", q);
 241             return retval;
 242         }
 243     }
 244
 245 Once the network port has been initialized using the correct VMDQ and DCB values,
 246 the initialization of the port's RX and TX hardware rings is performed similarly to that
 247 in the L2 Forwarding sample application.
 248 See :doc:`l2_forward_real_virtual` for more information.
 249
 250 Statistics Display
 251 ~~~~~~~~~~~~~~~~~~
 252
 253 When run in a linux environment,
 254 the VMDQ and DCB Forwarding sample application can display statistics showing the number of packets read from each RX queue.
 255 This is provided by way of a signal handler for the SIGHUP signal,
 256 which simply prints to standard output the packet counts in grid form.
 257 Each row of the output is a single pool with the columns being the queue number within that pool.
 258
 259 To generate the statistics output, use the following command:
 260
 261 .. code-block:: console
 262
 263     user@host$ sudo killall -HUP vmdq_dcb_app
 264
 265 Please note that the statistics output will appear on the terminal where the vmdq_dcb_app is running,
 266 rather than the terminal from which the HUP signal was sent.