]>
Commit | Line | Data |
---|---|---|
8cc2aca4 WC |
1 | Cross-Compiling |
2 | =============== | |
3 | ||
4 | FRR is capable of being cross-compiled to a number of different architectures. | |
5 | With an adequate toolchain this process is fairly straightforward, though one | |
6 | must exercise caution to validate this toolchain's correctness before attempting | |
7 | to compile FRR or its dependencies; small oversights in the construction of the | |
8 | build tools may lead to problems which quickly become difficult to diagnose. | |
9 | ||
10 | Toolchain Preliminary | |
11 | --------------------- | |
12 | ||
13 | The first step to cross-compiling any program is to identify the system which | |
14 | the program (FRR) will run on. From here on this will be called the "host" | |
15 | machine, following autotools' convention, while the machine building FRR will be | |
16 | called the "build" machine. The toolchain will of course be installed onto the | |
17 | build machine and be leveraged to build FRR for the host machine to run. | |
18 | ||
19 | .. note:: | |
20 | ||
21 | The build machine used while writing this guide was ``x86_64-pc-linux-gnu`` | |
22 | and the target machine was ``arm-linux-gnueabihf`` (a Raspberry Pi 3B+). | |
23 | Replace this with your targeted tuple below if you plan on running the | |
24 | commands from this guide: | |
25 | ||
26 | .. code-block:: shell | |
27 | ||
28 | export HOST_ARCH="arm-linux-gnueabihf" | |
29 | ||
30 | For your given target, the build system's OS may have some support for | |
31 | building cross compilers natively, or may even offer binary toolchains built | |
32 | upstream for the target architecture. Check your package manager or OS | |
33 | documentation before committing to building a toolchain from scratch. | |
34 | ||
35 | This guide will not detail *how* to build a cross-compiling toolchain but | |
36 | will instead assume one already exists and is installed on the build system. | |
37 | The methods for building the toolchain itself may differ between operating | |
38 | systems so consult the OS documentation for any particulars regarding | |
39 | cross-compilers. The OSDev wiki has a `pleasant tutorial`_ on cross-compiling in | |
40 | the context of operating system development which bootstraps from only the | |
41 | native GCC and binutils on the build machine. This may be useful if the build | |
42 | machine's OS does not offer existing tools to build a cross-compiler targeting | |
43 | the host. | |
44 | ||
45 | .. _pleasant tutorial: https://wiki.osdev.org/GCC_Cross-Compiler | |
46 | ||
47 | This guide will also not demonstrate how to build all of FRR's dependencies for the | |
48 | target architecture. Instead, general instructions for using a cross-compiling | |
49 | toolchain to compile packages using CMake, Autotools, and Makefiles are | |
50 | provided; these three cases apply to almost all FRR dependencies. | |
51 | ||
52 | .. _glibc mismatch: | |
53 | ||
54 | .. warning:: | |
55 | ||
56 | Ensure the versions and implementations of the C standard library (glibc or | |
57 | what have you) match on the host and the build toolchain. ``ldd --version`` | |
58 | will help you here. Upgrade one or the other if the they do not match. | |
59 | ||
60 | Testing the Toolchain | |
61 | --------------------- | |
62 | ||
63 | Before any cross-compilation begins it would be prudent to test the new | |
64 | toolchain by writing, compiling and linking a simple program. | |
65 | ||
66 | .. code-block:: shell | |
67 | ||
68 | # A small program | |
69 | cat > nothing.c <<EOF | |
70 | int main() { return 0; } | |
71 | EOF | |
72 | ||
73 | # Build and link with the cross-compiler | |
74 | ${HOST_ARCH}-gcc -o nothing nothing.c | |
75 | ||
76 | # Inspect the resulting binary, results may vary | |
77 | file ./nothing | |
78 | ||
79 | # nothing: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), | |
80 | # dynamically linked, interpreter /lib/ld-linux-armhf.so.3, | |
81 | # for GNU/Linux 3.2.0, not stripped | |
82 | ||
83 | If this produced no errors then the installed toolchain is probably ready to | |
84 | start compiling the build dependencies and eventually FRR itself. There still | |
85 | may be lurking issues but fundamentally the toolchain can produce binaries and | |
86 | that's good enough to start working with it. | |
87 | ||
88 | .. warning:: | |
89 | ||
90 | If any errors occurred during the previous functional test please look back | |
91 | and address them before moving on; this indicates your cross-compiling | |
92 | toolchain is *not* in a position to build FRR or its dependencies. Even if | |
93 | everything was fine, keep in mind that many errors from here on *may still | |
94 | be related* to your toolchain (e.g. libstdc++.so or other components) and this | |
95 | small test is not a guarantee of complete toolchain coherence. | |
96 | ||
97 | Cross-compiling Dependencies | |
98 | ---------------------------- | |
99 | ||
100 | When compiling FRR it is necessary to compile some of its dependencies alongside | |
101 | it on the build machine. This is so symbols from the shared libraries (which | |
102 | will be loaded at run-time on the host machine) can be linked to the FRR | |
103 | binaries at compile time; additionally, headers for these libraries are needed | |
104 | during the compile stage for a successful build. | |
105 | ||
106 | Sysroot Overview | |
107 | ^^^^^^^^^^^^^^^^ | |
108 | ||
109 | All build dependencies should be installed into a "root" directory on the build | |
110 | computer, hereafter called the "sysroot". This directory will be prefixed to | |
111 | paths while searching for requisite libraries and headers during the build | |
112 | process. Often this may be set via a ``--prefix`` flag when building the | |
113 | dependent packages, meaning a ``make install`` will copy compiled libraries into | |
114 | (e.g.) ``/usr/${HOST_ARCH}/usr``. | |
115 | ||
116 | If the toolchain was built on the build machine then there is likely already a | |
117 | sysroot where those tools and standard libraries were installed; it may be | |
118 | helpful to use that directory as the sysroot for this build as well. | |
119 | ||
120 | Basic Workflow | |
121 | ^^^^^^^^^^^^^^ | |
122 | ||
123 | Before compiling or building any dependencies, make note of which daemons are | |
124 | being targeted and which libraries will be needed. Not all dependencies are | |
125 | necessary if only building with a subset of the daemons. | |
126 | ||
127 | The following workflow will compile and install any libraries which can be built | |
128 | with Autotools. The resultant library will be installed into the sysroot | |
129 | ``/usr/${HOST_ARCH}``. | |
130 | ||
131 | .. code-block:: shell | |
132 | ||
133 | ./configure \ | |
134 | CC=${HOST_ARCH}-gcc \ | |
135 | CXX=${HOST_ARCH}-g++ \ | |
136 | --build=${HOST_ARCH} \ | |
137 | --prefix=/usr/${HOST_ARCH} | |
138 | make | |
139 | make install | |
140 | ||
141 | Some libraries like ``json-c`` and ``libyang`` are packaged with CMake and can | |
142 | be built and installed generally like: | |
143 | ||
144 | .. code-block:: shell | |
145 | ||
146 | mkdir build | |
147 | cd build | |
148 | CC=${HOST_ARCH}-gcc \ | |
149 | CXX=${HOST_ARCH}-g++ \ | |
150 | cmake \ | |
151 | -DCMAKE_INSTALL_PREFIX=/usr/${HOST_ARCH} \ | |
152 | .. | |
153 | make | |
154 | make install | |
155 | ||
156 | For programs with only a Makefile (e.g. ``libcap``) the process may look still a | |
157 | little different: | |
158 | ||
159 | .. code-block:: shell | |
160 | ||
161 | CC=${HOST_ARCH}-gcc make | |
162 | make install DESTDIR=/usr/${HOST_ARCH} | |
163 | ||
164 | These three workflows should handle the bulk of building and installing the | |
165 | build-time dependencies for FRR. Verify that the installed files are being | |
166 | placed correctly into the sysroot and were actually built using the | |
167 | cross-compile toolchain, not by the native toolchain by accident. | |
168 | ||
169 | Dependency Notes | |
170 | ^^^^^^^^^^^^^^^^ | |
171 | ||
172 | There are a lot of things that can go wrong during a cross-compilation. Some of | |
173 | the more common errors and a few special considerations are collected below for | |
174 | reference. | |
175 | ||
176 | libyang | |
177 | """"""" | |
178 | ||
179 | ``-DENABLE_LYD_PRIV=ON`` should be provided during the CMake step. | |
180 | ||
181 | Ensure also that the version of ``libyang`` being installed corresponds to the | |
182 | version required by the targeted FRR version. | |
183 | ||
184 | gRPC | |
185 | """" | |
186 | ||
187 | This piece is requisite only if the ``--enable-grpc`` flag will be passed | |
188 | later on to FRR. One may get burned when compiling gRPC if the ``protoc`` | |
189 | version on the build machine differs from the version of ``protoc`` being linked | |
190 | to during a gRPC build. The error messages from this defect look like: | |
191 | ||
4f32d35c | 192 | .. code-block:: shell |
8cc2aca4 WC |
193 | |
194 | gens/src/proto/grpc/channelz/channelz.pb.h: In member function ‘void grpc::channelz::v1::ServerRef::set_name(const char*, size_t)’: | |
195 | gens/src/proto/grpc/channelz/channelz.pb.h:9127:64: error: ‘EmptyDefault’ is not a member of ‘google::protobuf::internal::ArenaStringPtr’ | |
196 | 9127 | name_.Set(::PROTOBUF_NAMESPACE_ID::internal::ArenaStringPtr::EmptyDefault{}, ::std::string( | |
197 | ||
198 | This happens because protocol buffer code generation uses ``protoc`` to create | |
199 | classes with different getters and setters corresponding to the protobuf data | |
200 | defined by the source tree's ``.proto`` files. Clearly the cross-compiled | |
201 | ``protoc`` cannot be used for this code generation because that binary is built | |
202 | for a different CPU. | |
203 | ||
204 | The solution is to install matching versions of native and cross-compiled | |
205 | protocol buffers; this way the native binary will generate code and the | |
206 | cross-compiled library will be linked to by gRPC and these versions will not | |
207 | disagree. | |
208 | ||
209 | ---- | |
210 | ||
211 | The ``-latomic`` linker flag may also be necessary here if using ``libstdc++`` | |
212 | since GCC's C++11 implementation makes library calls in certain cases for | |
213 | ``<atomic>`` so ``-latomic`` cannot be assumed. | |
214 | ||
215 | Cross-compiling FRR Itself | |
216 | -------------------------- | |
217 | ||
218 | With all the necessary libraries cross-compiled and installed into the sysroot, | |
219 | the last thing to actually build is FRR itself: | |
220 | ||
221 | .. code-block:: shell | |
222 | ||
223 | # Clone and bootstrap the build | |
224 | git clone 'https://github.com/FRRouting/frr.git' | |
225 | # (e.g.) git checkout stable/7.5 | |
226 | ./bootstrap.sh | |
227 | ||
228 | # Build clippy using the native toolchain | |
229 | mkdir build-clippy | |
230 | cd build-clippy | |
231 | ../configure --enable-clippy-only | |
232 | make clippy-only | |
233 | cd .. | |
234 | ||
235 | # Next, configure FRR and use the clippy we just built | |
236 | ./configure \ | |
237 | CC=${HOST_ARCH}-gcc \ | |
238 | CXX=${HOST_ARCH}-g++ \ | |
239 | --host=${HOST_ARCH} \ | |
240 | --with-sysroot=/usr/${HOST_ARCH} \ | |
241 | --with-clippy=./build-clippy/lib/clippy \ | |
242 | --sysconfdir=/etc/frr \ | |
243 | --sbindir="\${prefix}/lib/frr" \ | |
244 | --localstatedir=/var/run/frr \ | |
245 | --prefix=/usr \ | |
246 | --enable-user=frr \ | |
247 | --enable-group=frr \ | |
248 | --enable-vty-group=frrvty \ | |
249 | --disable-doc \ | |
250 | --enable-grpc | |
251 | ||
252 | # Send it | |
253 | make | |
254 | ||
255 | Installation to Host Machine | |
256 | ---------------------------- | |
257 | ||
258 | If no errors were observed during the previous steps it is safe to ``make | |
259 | install`` FRR into its own directory. | |
260 | ||
261 | .. code-block:: shell | |
262 | ||
263 | # Install FRR its own "sysroot" | |
264 | make install DESTDIR=/some/path/to/sysroot | |
265 | ||
266 | After running the above command, FRR binaries, modules and example configuration | |
267 | files will be installed into some path on the build machine. The directory | |
268 | will have folders like ``/usr`` and ``/etc``; this "root" should now be copied | |
269 | to the host and installed on top of the root directory there. | |
270 | ||
271 | .. code-block:: shell | |
272 | ||
273 | # Tar this sysroot (preserving permissions) | |
274 | tar -C /some/path/to/sysroot -cpvf frr-${HOST_ARCH}.tar . | |
275 | ||
276 | # Transfer tar file to host machine | |
277 | scp frr-${HOST_ARCH}.tar me@host-machine: | |
278 | ||
279 | # Overlay the tarred sysroot on top of the host machine's root | |
280 | ssh me@host-machine <<-EOF | |
281 | # May need to elevate permissions here | |
282 | tar -C / -xpvf frr-${HOST_ARCH}.tar.gz . | |
283 | EOF | |
284 | ||
285 | Now FRR should be installed just as if ``make install`` had been run on the host | |
286 | machine. Create configuration files and assign permissions as needed. Lastly, | |
287 | ensure the correct users and groups exist for FRR on the host machine. | |
288 | ||
289 | Troubleshooting | |
290 | --------------- | |
291 | ||
292 | Even when every precaution has been taken some things may still go wrong! This | |
293 | section details some common runtime problems. | |
294 | ||
295 | Mismatched Libraries | |
296 | ^^^^^^^^^^^^^^^^^^^^ | |
297 | ||
298 | If you see something like this after installing on the host: | |
299 | ||
300 | .. code-block:: console | |
301 | ||
302 | /usr/lib/frr/zebra: error while loading shared libraries: libyang.so.1: cannot open shared object file: No such file or directory | |
303 | ||
304 | ... at least one of FRR's dependencies which was linked to the binary earlier is | |
305 | not available on the host OS. Even if it has been installed the host | |
306 | repository's version may lag what is needed for more recent FRR builds (this is | |
307 | likely to happen with YANG at the moment). | |
308 | ||
309 | If the matching library is not available from the host OS package manager it may | |
310 | be possible to compile them using the same toolchain used to compile FRR. The | |
311 | library may have already been built earlier when compiling FRR on the build | |
312 | machine, in which case it may be as simple as following the same workflow laid | |
313 | out during the `Installation to Host Machine`_. | |
314 | ||
315 | Mismatched Glibc Versions | |
316 | ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
317 | ||
318 | The version and implementation of the C standard library must match on both the | |
319 | host and build toolchain. The error corresponding to this misconfiguration will | |
320 | look like: | |
321 | ||
322 | .. code-block:: console | |
323 | ||
324 | /usr/lib/frr/zebra: /lib/${HOST_ARCH}/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libfrr.so.0) | |
325 | ||
326 | See the earlier warning about preventing a `glibc mismatch`_. |