4 FRR is capable of being cross-compiled to a number of different architectures.
5 With an adequate toolchain this process is fairly straightforward, though one
6 must exercise caution to validate this toolchain's correctness before attempting
7 to compile FRR or its dependencies; small oversights in the construction of the
8 build tools may lead to problems which quickly become difficult to diagnose.
13 The first step to cross-compiling any program is to identify the system which
14 the program (FRR) will run on. From here on this will be called the "host"
15 machine, following autotools' convention, while the machine building FRR will be
16 called the "build" machine. The toolchain will of course be installed onto the
17 build machine and be leveraged to build FRR for the host machine to run.
21 The build machine used while writing this guide was ``x86_64-pc-linux-gnu``
22 and the target machine was ``arm-linux-gnueabihf`` (a Raspberry Pi 3B+).
23 Replace this with your targeted tuple below if you plan on running the
24 commands from this guide:
28 export HOST_ARCH="arm-linux-gnueabihf"
30 For your given target, the build system's OS may have some support for
31 building cross compilers natively, or may even offer binary toolchains built
32 upstream for the target architecture. Check your package manager or OS
33 documentation before committing to building a toolchain from scratch.
35 This guide will not detail *how* to build a cross-compiling toolchain but
36 will instead assume one already exists and is installed on the build system.
37 The methods for building the toolchain itself may differ between operating
38 systems so consult the OS documentation for any particulars regarding
39 cross-compilers. The OSDev wiki has a `pleasant tutorial`_ on cross-compiling in
40 the context of operating system development which bootstraps from only the
41 native GCC and binutils on the build machine. This may be useful if the build
42 machine's OS does not offer existing tools to build a cross-compiler targeting
45 .. _pleasant tutorial: https://wiki.osdev.org/GCC_Cross-Compiler
47 This guide will also not demonstrate how to build all of FRR's dependencies for the
48 target architecture. Instead, general instructions for using a cross-compiling
49 toolchain to compile packages using CMake, Autotools, and Makefiles are
50 provided; these three cases apply to almost all FRR dependencies.
56 Ensure the versions and implementations of the C standard library (glibc or
57 what have you) match on the host and the build toolchain. ``ldd --version``
58 will help you here. Upgrade one or the other if the they do not match.
63 Before any cross-compilation begins it would be prudent to test the new
64 toolchain by writing, compiling and linking a simple program.
70 int main() { return 0; }
73 # Build and link with the cross-compiler
74 ${HOST_ARCH}-gcc -o nothing nothing.c
76 # Inspect the resulting binary, results may vary
79 # nothing: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV),
80 # dynamically linked, interpreter /lib/ld-linux-armhf.so.3,
81 # for GNU/Linux 3.2.0, not stripped
83 If this produced no errors then the installed toolchain is probably ready to
84 start compiling the build dependencies and eventually FRR itself. There still
85 may be lurking issues but fundamentally the toolchain can produce binaries and
86 that's good enough to start working with it.
90 If any errors occurred during the previous functional test please look back
91 and address them before moving on; this indicates your cross-compiling
92 toolchain is *not* in a position to build FRR or its dependencies. Even if
93 everything was fine, keep in mind that many errors from here on *may still
94 be related* to your toolchain (e.g. libstdc++.so or other components) and this
95 small test is not a guarantee of complete toolchain coherence.
97 Cross-compiling Dependencies
98 ----------------------------
100 When compiling FRR it is necessary to compile some of its dependencies alongside
101 it on the build machine. This is so symbols from the shared libraries (which
102 will be loaded at run-time on the host machine) can be linked to the FRR
103 binaries at compile time; additionally, headers for these libraries are needed
104 during the compile stage for a successful build.
109 All build dependencies should be installed into a "root" directory on the build
110 computer, hereafter called the "sysroot". This directory will be prefixed to
111 paths while searching for requisite libraries and headers during the build
112 process. Often this may be set via a ``--prefix`` flag when building the
113 dependent packages, meaning a ``make install`` will copy compiled libraries into
114 (e.g.) ``/usr/${HOST_ARCH}/usr``.
116 If the toolchain was built on the build machine then there is likely already a
117 sysroot where those tools and standard libraries were installed; it may be
118 helpful to use that directory as the sysroot for this build as well.
123 Before compiling or building any dependencies, make note of which daemons are
124 being targeted and which libraries will be needed. Not all dependencies are
125 necessary if only building with a subset of the daemons.
127 The following workflow will compile and install any libraries which can be built
128 with Autotools. The resultant library will be installed into the sysroot
129 ``/usr/${HOST_ARCH}``.
131 .. code-block:: shell
134 CC=${HOST_ARCH}-gcc \
135 CXX=${HOST_ARCH}-g++ \
136 --build=${HOST_ARCH} \
137 --prefix=/usr/${HOST_ARCH}
141 Some libraries like ``json-c`` and ``libyang`` are packaged with CMake and can
142 be built and installed generally like:
144 .. code-block:: shell
148 CC=${HOST_ARCH}-gcc \
149 CXX=${HOST_ARCH}-g++ \
151 -DCMAKE_INSTALL_PREFIX=/usr/${HOST_ARCH} \
156 For programs with only a Makefile (e.g. ``libcap``) the process may look still a
159 .. code-block:: shell
161 CC=${HOST_ARCH}-gcc make
162 make install DESTDIR=/usr/${HOST_ARCH}
164 These three workflows should handle the bulk of building and installing the
165 build-time dependencies for FRR. Verify that the installed files are being
166 placed correctly into the sysroot and were actually built using the
167 cross-compile toolchain, not by the native toolchain by accident.
172 There are a lot of things that can go wrong during a cross-compilation. Some of
173 the more common errors and a few special considerations are collected below for
179 ``-DENABLE_LYD_PRIV=ON`` should be provided during the CMake step.
181 Ensure also that the version of ``libyang`` being installed corresponds to the
182 version required by the targeted FRR version.
187 This piece is requisite only if the ``--enable-grpc`` flag will be passed
188 later on to FRR. One may get burned when compiling gRPC if the ``protoc``
189 version on the build machine differs from the version of ``protoc`` being linked
190 to during a gRPC build. The error messages from this defect look like:
192 .. code-block:: shell
194 gens/src/proto/grpc/channelz/channelz.pb.h: In member function ‘void grpc::channelz::v1::ServerRef::set_name(const char*, size_t)’:
195 gens/src/proto/grpc/channelz/channelz.pb.h:9127:64: error: ‘EmptyDefault’ is not a member of ‘google::protobuf::internal::ArenaStringPtr’
196 9127 | name_.Set(::PROTOBUF_NAMESPACE_ID::internal::ArenaStringPtr::EmptyDefault{}, ::std::string(
198 This happens because protocol buffer code generation uses ``protoc`` to create
199 classes with different getters and setters corresponding to the protobuf data
200 defined by the source tree's ``.proto`` files. Clearly the cross-compiled
201 ``protoc`` cannot be used for this code generation because that binary is built
204 The solution is to install matching versions of native and cross-compiled
205 protocol buffers; this way the native binary will generate code and the
206 cross-compiled library will be linked to by gRPC and these versions will not
211 The ``-latomic`` linker flag may also be necessary here if using ``libstdc++``
212 since GCC's C++11 implementation makes library calls in certain cases for
213 ``<atomic>`` so ``-latomic`` cannot be assumed.
215 Cross-compiling FRR Itself
216 --------------------------
218 With all the necessary libraries cross-compiled and installed into the sysroot,
219 the last thing to actually build is FRR itself:
221 .. code-block:: shell
223 # Clone and bootstrap the build
224 git clone 'https://github.com/FRRouting/frr.git'
225 # (e.g.) git checkout stable/7.5
228 # Build clippy using the native toolchain
231 ../configure --enable-clippy-only
235 # Next, configure FRR and use the clippy we just built
237 CC=${HOST_ARCH}-gcc \
238 CXX=${HOST_ARCH}-g++ \
239 --host=${HOST_ARCH} \
240 --with-sysroot=/usr/${HOST_ARCH} \
241 --with-clippy=./build-clippy/lib/clippy \
242 --sysconfdir=/etc/frr \
243 --sbindir="\${prefix}/lib/frr" \
244 --localstatedir=/var/run/frr \
248 --enable-vty-group=frrvty \
255 Installation to Host Machine
256 ----------------------------
258 If no errors were observed during the previous steps it is safe to ``make
259 install`` FRR into its own directory.
261 .. code-block:: shell
263 # Install FRR its own "sysroot"
264 make install DESTDIR=/some/path/to/sysroot
266 After running the above command, FRR binaries, modules and example configuration
267 files will be installed into some path on the build machine. The directory
268 will have folders like ``/usr`` and ``/etc``; this "root" should now be copied
269 to the host and installed on top of the root directory there.
271 .. code-block:: shell
273 # Tar this sysroot (preserving permissions)
274 tar -C /some/path/to/sysroot -cpvf frr-${HOST_ARCH}.tar .
276 # Transfer tar file to host machine
277 scp frr-${HOST_ARCH}.tar me@host-machine:
279 # Overlay the tarred sysroot on top of the host machine's root
280 ssh me@host-machine <<-EOF
281 # May need to elevate permissions here
282 tar -C / -xpvf frr-${HOST_ARCH}.tar.gz .
285 Now FRR should be installed just as if ``make install`` had been run on the host
286 machine. Create configuration files and assign permissions as needed. Lastly,
287 ensure the correct users and groups exist for FRR on the host machine.
292 Even when every precaution has been taken some things may still go wrong! This
293 section details some common runtime problems.
298 If you see something like this after installing on the host:
300 .. code-block:: console
302 /usr/lib/frr/zebra: error while loading shared libraries: libyang.so.1: cannot open shared object file: No such file or directory
304 ... at least one of FRR's dependencies which was linked to the binary earlier is
305 not available on the host OS. Even if it has been installed the host
306 repository's version may lag what is needed for more recent FRR builds (this is
307 likely to happen with YANG at the moment).
309 If the matching library is not available from the host OS package manager it may
310 be possible to compile them using the same toolchain used to compile FRR. The
311 library may have already been built earlier when compiling FRR on the build
312 machine, in which case it may be as simple as following the same workflow laid
313 out during the `Installation to Host Machine`_.
315 Mismatched Glibc Versions
316 ^^^^^^^^^^^^^^^^^^^^^^^^^
318 The version and implementation of the C standard library must match on both the
319 host and build toolchain. The error corresponding to this misconfiguration will
322 .. code-block:: console
324 /usr/lib/frr/zebra: /lib/${HOST_ARCH}/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libfrr.so.0)
326 See the earlier warning about preventing a `glibc mismatch`_.