]>
Commit | Line | Data |
---|---|---|
11fdf7f2 TL |
1 | .. SPDX-License-Identifier: BSD-3-Clause |
2 | Copyright(c) 2010-2014 Intel Corporation. | |
7c673cae FG |
3 | |
4 | .. _building_from_source: | |
5 | ||
6 | Compiling the DPDK Target from Source | |
7 | ===================================== | |
8 | ||
f67539c2 TL |
9 | Prerequisites |
10 | ------------- | |
7c673cae | 11 | |
f67539c2 | 12 | The following FreeBSD packages are required to build DPDK: |
7c673cae | 13 | |
f67539c2 TL |
14 | * meson |
15 | * ninja | |
16 | * pkgconf | |
7c673cae | 17 | |
f67539c2 | 18 | These can be installed using (as root):: |
7c673cae | 19 | |
f67539c2 | 20 | pkg install meson pkgconf |
7c673cae | 21 | |
f67539c2 TL |
22 | To compile the required kernel modules for memory management and working |
23 | with physical NIC devices, the kernel sources for FreeBSD also | |
24 | need to be installed. If not already present on the system, these can be | |
25 | installed via commands like the following, for FreeBSD 12.1 on x86_64:: | |
7c673cae | 26 | |
f67539c2 TL |
27 | fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/src.txz |
28 | tar -C / -xJvf src.txz | |
7c673cae | 29 | |
f67539c2 TL |
30 | To enable the telemetry library in DPDK, the jansson library also needs to |
31 | be installed, and can be installed via:: | |
7c673cae | 32 | |
f67539c2 | 33 | pkg install jansson |
7c673cae | 34 | |
f67539c2 TL |
35 | Individual drivers may have additional requirements. Consult the relevant |
36 | driver guide for any driver-specific requirements of interest. | |
7c673cae | 37 | |
f67539c2 TL |
38 | Building DPDK |
39 | ------------- | |
7c673cae | 40 | |
f67539c2 TL |
41 | The following commands can be used to build and install DPDK on a system. |
42 | The final, install, step generally needs to be run as root:: | |
7c673cae | 43 | |
f67539c2 TL |
44 | meson build |
45 | cd build | |
46 | ninja | |
47 | ninja install | |
7c673cae | 48 | |
f67539c2 TL |
49 | This will install the DPDK libraries and drivers to `/usr/local/lib` with a |
50 | pkg-config file `libdpdk.pc` installed to `/usr/local/lib/pkgconfig`. The | |
51 | DPDK test applications, such as `dpdk-testpmd` are installed to | |
52 | `/usr/local/bin`. To use these applications, it is recommended that the | |
53 | `contigmem` and `nic_uio` kernel modules be loaded first, as described in | |
54 | the next section. | |
7c673cae FG |
55 | |
56 | .. note:: | |
57 | ||
f67539c2 TL |
58 | It is recommended that pkg-config be used to query information |
59 | about the compiler and linker flags needed to build applications | |
60 | against DPDK. In some cases, the path `/usr/local/lib/pkgconfig` | |
61 | may not be in the default search paths for `.pc` files, which means | |
62 | that queries for DPDK information may fail. This can be fixed by | |
63 | setting the appropriate path in `PKG_CONFIG_PATH` environment | |
64 | variable. | |
7c673cae | 65 | |
7c673cae FG |
66 | |
67 | .. _loading_contigmem: | |
68 | ||
69 | Loading the DPDK contigmem Module | |
70 | --------------------------------- | |
71 | ||
72 | To run a DPDK application, physically contiguous memory is required. | |
73 | In the absence of non-transparent superpages, the included sources for the | |
74 | contigmem kernel module provides the ability to present contiguous blocks of | |
75 | memory for the DPDK to use. The contigmem module must be loaded into the | |
f67539c2 TL |
76 | running kernel before any DPDK is run. Once DPDK is installed on the |
77 | system, the module can be found in the `/boot/modules` directory. | |
7c673cae FG |
78 | |
79 | The amount of physically contiguous memory along with the number of physically | |
80 | contiguous blocks to be reserved by the module can be set at runtime prior to | |
f67539c2 | 81 | module loading using:: |
7c673cae FG |
82 | |
83 | kenv hw.contigmem.num_buffers=n | |
84 | kenv hw.contigmem.buffer_size=m | |
85 | ||
86 | The kernel environment variables can also be specified during boot by placing the | |
f67539c2 | 87 | following in ``/boot/loader.conf``: |
7c673cae | 88 | |
f67539c2 | 89 | .. code-block:: shell |
7c673cae | 90 | |
f67539c2 TL |
91 | hw.contigmem.num_buffers=n |
92 | hw.contigmem.buffer_size=m | |
7c673cae | 93 | |
f67539c2 | 94 | The variables can be inspected using the following command:: |
7c673cae FG |
95 | |
96 | sysctl -a hw.contigmem | |
97 | ||
98 | Where n is the number of blocks and m is the size in bytes of each area of | |
99 | contiguous memory. A default of two buffers of size 1073741824 bytes (1 Gigabyte) | |
100 | each is set during module load if they are not specified in the environment. | |
101 | ||
f67539c2 | 102 | The module can then be loaded using kldload:: |
7c673cae | 103 | |
f67539c2 | 104 | kldload contigmem |
7c673cae FG |
105 | |
106 | It is advisable to include the loading of the contigmem module during the boot | |
107 | process to avoid issues with potential memory fragmentation during later system | |
f67539c2 TL |
108 | up time. This can be achieved by placing lines similar to the following into |
109 | ``/boot/loader.conf``: | |
110 | ||
111 | .. code-block:: shell | |
7c673cae | 112 | |
f67539c2 TL |
113 | hw.contigmem.num_buffers=1 |
114 | hw.contigmem.buffer_size=1073741824 | |
7c673cae FG |
115 | contigmem_load="YES" |
116 | ||
117 | .. note:: | |
118 | ||
119 | The contigmem_load directive should be placed after any definitions of | |
120 | ``hw.contigmem.num_buffers`` and ``hw.contigmem.buffer_size`` if the default values | |
121 | are not to be used. | |
122 | ||
f67539c2 | 123 | An error such as:: |
7c673cae | 124 | |
9f95a23c | 125 | kldload: can't load ./x86_64-native-freebsd-gcc/kmod/contigmem.ko: |
7c673cae FG |
126 | Exec format error |
127 | ||
128 | is generally attributed to not having enough contiguous memory | |
f67539c2 | 129 | available and can be verified via dmesg or ``/var/log/messages``:: |
7c673cae FG |
130 | |
131 | kernel: contigmalloc failed for buffer <n> | |
132 | ||
133 | To avoid this error, reduce the number of buffers or the buffer size. | |
134 | ||
135 | .. _loading_nic_uio: | |
136 | ||
137 | Loading the DPDK nic_uio Module | |
138 | ------------------------------- | |
139 | ||
140 | After loading the contigmem module, the ``nic_uio`` module must also be loaded into the | |
f67539c2 | 141 | running kernel prior to running any DPDK application, e.g. using:: |
7c673cae | 142 | |
f67539c2 | 143 | kldload nic_uio |
7c673cae FG |
144 | |
145 | .. note:: | |
146 | ||
147 | If the ports to be used are currently bound to a existing kernel driver | |
148 | then the ``hw.nic_uio.bdfs sysctl`` value will need to be set before loading the | |
149 | module. Setting this value is described in the next section below. | |
150 | ||
151 | Currently loaded modules can be seen by using the ``kldstat`` command and a module | |
152 | can be removed from the running kernel by using ``kldunload <module_name>``. | |
153 | ||
f67539c2 TL |
154 | To load the module during boot place the following into ``/boot/loader.conf``: |
155 | ||
156 | .. code-block:: shell | |
7c673cae FG |
157 | |
158 | nic_uio_load="YES" | |
159 | ||
160 | .. note:: | |
161 | ||
162 | ``nic_uio_load="YES"`` must appear after the contigmem_load directive, if it exists. | |
163 | ||
164 | By default, the ``nic_uio`` module will take ownership of network ports if they are | |
165 | recognized DPDK devices and are not owned by another module. However, since | |
166 | the FreeBSD kernel includes support, either built-in, or via a separate driver | |
167 | module, for most network card devices, it is likely that the ports to be used are | |
168 | already bound to a driver other than ``nic_uio``. The following sub-section describe | |
169 | how to query and modify the device ownership of the ports to be used by | |
170 | DPDK applications. | |
171 | ||
172 | .. _binding_network_ports: | |
173 | ||
174 | Binding Network Ports to the nic_uio Module | |
175 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
176 | ||
177 | Device ownership can be viewed using the pciconf -l command. The example below shows | |
178 | four Intel® 82599 network ports under ``if_ixgbe`` module ownership. | |
179 | ||
f67539c2 | 180 | .. code-block:: none |
7c673cae FG |
181 | |
182 | pciconf -l | |
183 | ix0@pci0:1:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00 | |
184 | ix1@pci0:1:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00 | |
185 | ix2@pci0:2:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00 | |
186 | ix3@pci0:2:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00 | |
187 | ||
188 | The first column constitutes three components: | |
189 | ||
190 | #. Device name: ``ixN`` | |
191 | ||
192 | #. Unit name: ``pci0`` | |
193 | ||
194 | #. Selector (Bus:Device:Function): ``1:0:0`` | |
195 | ||
196 | Where no driver is associated with a device, the device name will be ``none``. | |
197 | ||
198 | By default, the FreeBSD kernel will include built-in drivers for the most common | |
199 | devices; a kernel rebuild would normally be required to either remove the drivers | |
200 | or configure them as loadable modules. | |
201 | ||
202 | To avoid building a custom kernel, the ``nic_uio`` module can detach a network port | |
203 | from its current device driver. This is achieved by setting the ``hw.nic_uio.bdfs`` | |
204 | kernel environment variable prior to loading ``nic_uio``, as follows:: | |
205 | ||
f67539c2 | 206 | kenv hw.nic_uio.bdfs="b:d:f,b:d:f,..." |
7c673cae FG |
207 | |
208 | Where a comma separated list of selectors is set, the list must not contain any | |
209 | whitespace. | |
210 | ||
211 | For example to re-bind ``ix2@pci0:2:0:0`` and ``ix3@pci0:2:0:1`` to the ``nic_uio`` module | |
212 | upon loading, use the following command:: | |
213 | ||
214 | kenv hw.nic_uio.bdfs="2:0:0,2:0:1" | |
215 | ||
216 | The variable can also be specified during boot by placing the following into | |
217 | ``/boot/loader.conf``, before the previously-described ``nic_uio_load`` line - as | |
f67539c2 TL |
218 | shown: |
219 | ||
220 | .. code-block:: shell | |
7c673cae FG |
221 | |
222 | hw.nic_uio.bdfs="2:0:0,2:0:1" | |
223 | nic_uio_load="YES" | |
224 | ||
225 | Binding Network Ports Back to their Original Kernel Driver | |
226 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
227 | ||
228 | If the original driver for a network port has been compiled into the kernel, | |
229 | it is necessary to reboot FreeBSD to restore the original device binding. Before | |
230 | doing so, update or remove the ``hw.nic_uio.bdfs`` in ``/boot/loader.conf``. | |
231 | ||
232 | If rebinding to a driver that is a loadable module, the network port binding can | |
233 | be reset without rebooting. To do so, unload both the target kernel module and the | |
234 | ``nic_uio`` module, modify or clear the ``hw.nic_uio.bdfs`` kernel environment (kenv) | |
235 | value, and reload the two drivers - first the original kernel driver, and then | |
236 | the ``nic_uio driver``. Note: the latter does not need to be reloaded unless there are | |
237 | ports that are still to be bound to it. | |
238 | ||
f67539c2 | 239 | Example commands to perform these steps are shown below:: |
7c673cae FG |
240 | |
241 | kldunload nic_uio | |
242 | kldunload <original_driver> | |
243 | ||
244 | # To clear the value completely: | |
245 | kenv -u hw.nic_uio.bdfs | |
246 | ||
247 | # To update the list of ports to bind: | |
248 | kenv hw.nic_uio.bdfs="b:d:f,b:d:f,..." | |
249 | ||
250 | kldload <original_driver> | |
251 | ||
252 | kldload nic_uio # optional |