ceph/src/spdk/dpdk/doc/guides/sample_app_ug/l3_forward_graph.rst

   1 ..  SPDX-License-Identifier: BSD-3-Clause
   2     Copyright(C) 2020 Marvell International Ltd.
   3
   4 L3 Forwarding Graph Sample Application
   5 ======================================
   6
   7 The L3 Forwarding Graph application is a simple example of packet processing
   8 using the DPDK Graph framework. The application performs L3 forwarding using
   9 Graph framework and nodes written for graph framework.
  10
  11 Overview
  12 --------
  13
  14 The application demonstrates the use of the graph framework and graph nodes
  15 ``ethdev_rx``, ``ip4_lookup``, ``ip4_rewrite``, ``ethdev_tx`` and ``pkt_drop`` in DPDK to
  16 implement packet forwarding.
  17
  18 The initialization is very similar to those of the :doc:`l3_forward`.
  19 There is also additional initialization of graph for graph object creation
  20 and configuration per lcore.
  21 Run-time path is main thing that differs from L3 forwarding sample application.
  22 Difference is that forwarding logic starting from Rx, followed by LPM lookup,
  23 TTL update and finally Tx is implemented inside graph nodes. These nodes are
  24 interconnected in graph framework. Application main loop needs to walk over
  25 graph using ``rte_graph_walk()`` with graph objects created one per slave lcore.
  26
  27 The lookup method is as per implementation of ``ip4_lookup`` graph node.
  28 The ID of the output interface for the input packet is the next hop returned by
  29 the LPM lookup. The set of LPM rules used by the application is statically
  30 configured and provided to ``ip4_lookup`` graph node and ``ip4_rewrite`` graph node
  31 using node control API ``rte_node_ip4_route_add()`` and ``rte_node_ip4_rewrite_add()``.
  32
  33 In the sample application, only IPv4 forwarding is supported as of now.
  34
  35 Compiling the Application
  36 -------------------------
  37
  38 To compile the sample application see :doc:`compiling`.
  39
  40 The application is located in the ``l3fwd-graph`` sub-directory.
  41
  42 Running the Application
  43 -----------------------
  44
  45 The application has a number of command line options similar to l3fwd::
  46
  47     ./l3fwd-graph [EAL options] -- -p PORTMASK
  48                                    [-P]
  49                                    --config(port,queue,lcore)[,(port,queue,lcore)]
  50                                    [--eth-dest=X,MM:MM:MM:MM:MM:MM]
  51                                    [--enable-jumbo [--max-pkt-len PKTLEN]]
  52                                    [--no-numa]
  53                                    [--per-port-pool]
  54
  55 Where,
  56
  57 * ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure
  58
  59 * ``-P:`` Optional, sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
  60   Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
  61
  62 * ``--config (port,queue,lcore)[,(port,queue,lcore)]:`` Determines which queues from which ports are mapped to which cores.
  63
  64 * ``--eth-dest=X,MM:MM:MM:MM:MM:MM:`` Optional, ethernet destination for port X.
  65
  66 * ``--enable-jumbo:`` Optional, enables jumbo frames.
  67
  68 * ``--max-pkt-len:`` Optional, under the premise of enabling jumbo, maximum packet length in decimal (64-9600).
  69
  70 * ``--no-numa:`` Optional, disables numa awareness.
  71
  72 * ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports.
  73
  74 For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0,
  75 while cores 8-15 and 24-31 appear on socket 1.
  76
  77 To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2,
  78 (which are in the same socket too), use the following command:
  79
  80 .. code-block:: console
  81
  82     ./build/l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)"
  83
  84 In this command:
  85
  86 *   The -l option enables cores 1, 2
  87
  88 *   The -p option enables ports 0 and 1
  89
  90 *   The --config option enables one queue on each port and maps each (port,queue) pair to a specific core.
  91     The following table shows the mapping in this example:
  92
  93 +----------+-----------+-----------+-------------------------------------+
  94 | **Port** | **Queue** | **lcore** | **Description**                     |
  95 |          |           |           |                                     |
  96 +----------+-----------+-----------+-------------------------------------+
  97 | 0        | 0         | 1         | Map queue 0 from port 0 to lcore 1. |
  98 |          |           |           |                                     |
  99 +----------+-----------+-----------+-------------------------------------+
 100 | 1        | 0         | 2         | Map queue 0 from port 1 to lcore 2. |
 101 |          |           |           |                                     |
 102 +----------+-----------+-----------+-------------------------------------+
 103
 104 Refer to the *DPDK Getting Started Guide* for general information on running applications and
 105 the Environment Abstraction Layer (EAL) options.
 106
 107 .. _l3_fwd_graph_explanation:
 108
 109 Explanation
 110 -----------
 111
 112 The following sections provide some explanation of the sample application code.
 113 As mentioned in the overview section, the initialization is similar to that of
 114 the :doc:`l3_forward`. Run-time path though similar in functionality to that of
 115 :doc:`l3_forward`, major part of the implementation is in graph nodes via used
 116 via ``librte_node`` library.
 117 The following sections describe aspects that are specific to the L3 Forwarding
 118 Graph sample application.
 119
 120 Graph Node Pre-Init Configuration
 121 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 122
 123 After device configuration and device Rx, Tx queue setup is complete,
 124 a minimal config of port id, num_rx_queues, num_tx_queues, mempools etc will
 125 be passed to *ethdev_** node ctrl API ``rte_node_eth_config()``. This will be
 126 lead to the clone of ``ethdev_rx`` and ``ethdev_tx`` nodes as ``ethdev_rx-X-Y`` and
 127 ``ethdev_tx-X`` where X, Y represent port id and queue id associated with them.
 128 In case of ``ethdev_tx-X`` nodes, tx queue id assigned per instance of the node
 129 is same as graph id.
 130
 131 These cloned nodes along with existing static nodes such as ``ip4_lookup`` and
 132 ``ip4_rewrite`` will be used in graph creation to associate node's to lcore
 133 specific graph object.
 134
 135 .. code-block:: c
 136
 137     RTE_ETH_FOREACH_DEV(portid)
 138     {
 139
 140         /* ... */
 141         ret = rte_eth_dev_configure(portid, nb_rx_queue,
 142                                     n_tx_queue, &local_port_conf);
 143         /* ... */
 144
 145         /* Init one TX queue per couple (lcore,port) */
 146         queueid = 0;
 147         for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
 148             /* ... */
 149             ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd,
 150                                          socketid, txconf);
 151             /* ... */
 152             queueid++;
 153         }
 154
 155         /* Setup ethdev node config */
 156         ethdev_conf[nb_conf].port_id = portid;
 157         ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue;
 158         ethdev_conf[nb_conf].num_tx_queues = n_tx_queue;
 159         if (!per_port_pool)
 160             ethdev_conf[nb_conf].mp = pktmbuf_pool[0];
 161         else
 162           ethdev_conf[nb_conf].mp = pktmbuf_pool[portid];
 163         ethdev_conf[nb_conf].mp_count = NB_SOCKETS;
 164
 165         nb_conf++;
 166         printf("\n");
 167     }
 168
 169     for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
 170         /* Init RX queues */
 171         for (queue = 0; queue < qconf->n_rx_queue; ++queue) {
 172             /* ... */
 173             if (!per_port_pool)
 174                 ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, socketid,
 175                                              &rxq_conf, pktmbuf_pool[0][socketid]);
 176             else
 177               ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, socketid,
 178                                            &rxq_conf, pktmbuf_pool[portid][socketid]);
 179             /* ... */
 180         }
 181     }
 182
 183     /* Ethdev node config, skip rx queue mapping */
 184     ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs);
 185
 186 Graph Initialization
 187 ~~~~~~~~~~~~~~~~~~~~
 188
 189 Now a graph needs to be created with a specific set of nodes for every lcore.
 190 A graph object returned after graph creation is a per lcore object and
 191 cannot be shared between lcores. Since ``ethdev_tx-X`` node is per port node,
 192 it can be associated with all the graphs created as all the lcores should have
 193 Tx capability for every port. But ``ethdev_rx-X-Y`` node is created per
 194 (port, rx_queue_id), so they should be associated with a graph based on
 195 the application argument ``--config`` specifying rx queue mapping to lcore.
 196
 197 .. note::
 198
 199     The Graph creation will fail if the passed set of shell node pattern's
 200     are not sufficient to meet their inter-dependency or even one node is not
 201     found with a given regex node pattern.
 202
 203 .. code-block:: c
 204
 205     static const char *const default_patterns[] = {
 206         "ip4*",
 207         "ethdev_tx-*",
 208         "pkt_drop",
 209     };
 210     const char **node_patterns;
 211     uint16_t nb_pattern;
 212
 213     /* ... */
 214
 215     /* Create a graph object per lcore with common nodes and
 216      * lcore specific nodes based on application arguments
 217      */
 218     nb_patterns = RTE_DIM(default_patterns);
 219     node_patterns = malloc((MAX_RX_QUEUE_PER_LCORE + nb_patterns) *
 220                            sizeof(*node_patterns));
 221     memcpy(node_patterns, default_patterns,
 222            nb_patterns * sizeof(*node_patterns));
 223
 224     memset(&graph_conf, 0, sizeof(graph_conf));
 225
 226     /* Common set of nodes in every lcore's graph object */
 227     graph_conf.node_patterns = node_patterns;
 228
 229     for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
 230         /* ... */
 231
 232         /* Skip graph creation if no source exists */
 233         if (!qconf->n_rx_queue)
 234             continue;
 235
 236         /* Add rx node patterns of this lcore based on --config */
 237         for (i = 0; i < qconf->n_rx_queue; i++) {
 238             graph_conf.node_patterns[nb_patterns + i] =
 239                                 qconf->rx_queue_list[i].node_name;
 240         }
 241
 242         graph_conf.nb_node_patterns = nb_patterns + i;
 243         graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
 244
 245         snprintf(qconf->name, sizeof(qconf->name), "worker_%u", lcore_id);
 246
 247         graph_id = rte_graph_create(qconf->name, &graph_conf);
 248
 249         /* ... */
 250
 251         qconf->graph = rte_graph_lookup(qconf->name);
 252
 253         /* ... */
 254     }
 255
 256 Forwarding data(Route, Next-Hop) addition
 257 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 258
 259 Once graph objects are created, node specific info like routes and rewrite
 260 headers will be provided run-time using ``rte_node_ip4_route_add()`` and
 261 ``rte_node_ip4_rewrite_add()`` API.
 262
 263 .. note::
 264
 265     Since currently ``ip4_lookup`` and ``ip4_rewrite`` nodes don't support
 266     lock-less mechanisms(RCU, etc) to add run-time forwarding data like route and
 267     rewrite data, forwarding data is added before packet processing loop is
 268     launched on slave lcore.
 269
 270 .. code-block:: c
 271
 272     /* Add route to ip4 graph infra */
 273     for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
 274         /* ... */
 275
 276         dst_port = ipv4_l3fwd_lpm_route_array[i].if_out;
 277         next_hop = i;
 278
 279         /* ... */
 280         ret = rte_node_ip4_route_add(ipv4_l3fwd_lpm_route_array[i].ip,
 281                                      ipv4_l3fwd_lpm_route_array[i].depth, next_hop,
 282                                      RTE_NODE_IP4_LOOKUP_NEXT_REWRITE);
 283
 284         /* ... */
 285
 286         memcpy(rewrite_data, val_eth + dst_port, rewrite_len);
 287
 288         /* Add next hop for a given destination */
 289         ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data,
 290                                        rewrite_len, dst_port);
 291
 292         RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n",
 293                 route_str, next_hop);
 294     }
 295
 296 Packet Forwarding using Graph Walk
 297 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 298
 299 Now that all the device configurations are done, graph creations are done and
 300 forwarding data is updated with nodes, slave lcores will be launched with graph
 301 main loop. Graph main loop is very simple in the sense that it needs to
 302 continuously call a non-blocking API ``rte_graph_walk()`` with it's lcore
 303 specific graph object that was already created.
 304
 305 .. note::
 306
 307     rte_graph_walk() will walk over all the sources nodes i.e ``ethdev_rx-X-Y``
 308     associated with a given graph and Rx the available packets and enqueue them
 309     to the following node ``ip4_lookup`` which then will enqueue them to ``ip4_rewrite``
 310     node if LPM lookup succeeds. ``ip4_rewrite`` node then will update Ethernet header
 311     as per next-hop data and transmit the packet via port 'Z' by enqueuing
 312     to ``ethdev_tx-Z`` node instance in its graph object.
 313
 314 .. code-block:: c
 315
 316     /* Main processing loop */
 317     static int
 318     graph_main_loop(void *conf)
 319     {
 320         // ...
 321
 322         lcore_id = rte_lcore_id();
 323         qconf = &lcore_conf[lcore_id];
 324         graph = qconf->graph;
 325
 326         RTE_LOG(INFO, L3FWD_GRAPH,
 327                 "Entering main loop on lcore %u, graph %s(%p)\n", lcore_id,
 328                 qconf->name, graph);
 329
 330         /* Walk over graph until signal to quit */
 331         while (likely(!force_quit))
 332             rte_graph_walk(graph);
 333         return 0;
 334     }