6 Topotests is a suite of topology tests for FRR built on top of micronet.
11 Topotests run under python3. Additionally, for ExaBGP (which is used
12 in some of the BGP tests) an older python2 version (and the python2
13 version of ``pip``) must be installed.
15 Tested with Ubuntu 20.04,Ubuntu 18.04, and Debian 11.
17 Instructions are the same for all setups (i.e. ExaBGP is only used for
20 Installing Topotest Requirements
21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
26 apt-get install iproute2
27 apt-get install net-tools
28 apt-get install python3-pip
29 python3 -m pip install wheel
30 python3 -m pip install 'pytest>=6.2.4'
31 python3 -m pip install 'pytest-xdist>=2.3.0'
32 python3 -m pip install 'scapy>=2.4.5'
33 python3 -m pip install xmltodict
34 # Use python2 pip to install older ExaBGP
35 python2 -m pip install 'exabgp<4.0.0'
36 useradd -d /var/run/exabgp/ -s /bin/false exabgp
38 # To enable the gRPC topotest install:
39 python3 -m pip install grpcio grpcio-tools
41 # Install Socat tool to run PIMv6 tests,
42 # Socat code can be taken from below url,
43 # which has latest changes done for PIMv6,
45 https://github.com/opensourcerouting/socat/
51 Optional, will give better output.
55 disable apport (which move core files)
57 Set ``enabled=0`` in ``/etc/default/apport``.
59 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
61 #<domain> <type> <item> <value>
63 root soft core unlimited
65 root hard core unlimited
67 Reboot for options to take effect.
69 SNMP Utilities Installation
70 """""""""""""""""""""""""""
72 To run SNMP test you need to install SNMP utilities and MIBs. Unfortunately
73 there are some errors in the upstream MIBS which need to be patched up. The
74 following steps will get you there on Ubuntu 20.04.
78 apt install libsnmp-dev
79 apt install snmpd snmp
80 apt install snmp-mibs-downloader
82 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/iana/IANA-IPPM-METRICS-REGISTRY-MIB -O /usr/share/snmp/mibs/iana/IANA-IPPM-METRICS-REGISTRY-MIB
83 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/ietf/SNMPv2-PDU -O /usr/share/snmp/mibs/ietf/SNMPv2-PDU
84 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/ietf/IPATM-IPMC-MIB -O /usr/share/snmp/mibs/ietf/IPATM-IPMC-MIB
85 edit /etc/snmp/snmp.conf to look like this
86 # As the snmp packages come without MIB files due to license reasons, loading
87 # of MIBs is disabled by default. If you added the MIBs you can reenable
88 # loading them by commenting out the following line.
95 FRR needs to be installed separately. It is assume to be configured like the
96 standard Ubuntu Packages:
98 - Binaries in :file:`/usr/lib/frr`
99 - State Directory :file:`/var/run/frr`
100 - Running under user ``frr``, group ``frr``
101 - vtygroup: ``frrvty``
102 - config directory: :file:`/etc/frr`
103 - For FRR Packages, install the dbg package as well for coredump decoding
105 No FRR config needs to be done and no FRR daemons should be run ahead of the
106 test. They are all started as part of the test.
111 If you prefer to manually build FRR, then use the following suggested config:
117 --localstatedir=/var/run/frr \
118 --sbindir=/usr/lib/frr \
119 --sysconfdir=/etc/frr \
123 --enable-multipath=64 \
126 --enable-vty-group=frrvty \
127 --enable-snmp=agentx \
128 --with-pkg-extra-version=-my-manual-build
130 And create ``frr`` user and ``frrvty`` group as follows:
134 addgroup --system --gid 92 frr
135 addgroup --system --gid 85 frrvty
136 adduser --system --ingroup frr --home /var/run/frr/ \
137 --gecos "FRRouting suite" --shell /bin/false frr
138 usermod -G frrvty frr
143 Configure your sudo environment
144 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
146 Topotests must be run as root. Normally this will be accomplished through the
147 use of the ``sudo`` command. In order for topotests to be able to open new
148 windows (either XTerm or byobu/screen/tmux windows) certain environment
149 variables must be passed through the sudo command. One way to do this is to
150 specify the ``-E`` flag to ``sudo``. This will carry over most if not all
151 your environment variables include ``PATH``. For example:
155 sudo -E python3 -m pytest -s -v
157 If you do not wish to use ``-E`` (e.g., to avoid ``sudo`` inheriting
158 ``PATH``) you can modify your `/etc/sudoers` config file to specifically pass
159 the environment variables required by topotests. Add the following commands to
160 your ``/etc/sudoers`` config file.
164 Defaults env_keep="TMUX"
165 Defaults env_keep+="TMUX_PANE"
166 Defaults env_keep+="STY"
167 Defaults env_keep+="DISPLAY"
169 If there was already an ``env_keep`` configuration there be sure to use the
170 ``+=`` rather than ``=`` on the first line above as well.
173 Execute all tests in distributed test mode
174 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
178 sudo -E pytest -s -v -nauto --dist=loadfile
180 The above command must be executed from inside the topotests directory.
182 All test\_\* scripts in subdirectories are detected and executed (unless
183 disabled in ``pytest.ini`` file). Pytest will execute up to N tests in parallel
184 where N is based on the number of cores on the host.
186 Analyze Test Results (``analyze.py``)
187 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
189 By default router and execution logs are saved in ``/tmp/topotests`` and an XML
190 results file is saved in ``/tmp/topotests.xml``. An analysis tool ``analyze.py``
191 is provided to archive and analyze these results after the run completes.
193 After the test run completes one should pick an archive directory to store the
194 results in and pass this value to ``analyze.py``. On first execution the results
195 are copied to that directory from ``/tmp``, and subsequent runs use that
196 directory for analyzing the results. Below is an example of this which also
197 shows the default behavior which is to display all failed and errored tests in
202 ~/frr/tests/topotests# ./analyze.py -Ar run-save
203 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge
204 ospf_basic_functionality/test_ospf_lan.py::test_ospf_lan_tc1_p0
205 bgp_gr_functionality_topo2/test_bgp_gr_functionality_topo2.py::test_BGP_GR_10_p2
206 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_routingTable
208 Here we see that 4 tests have failed. We an dig deeper by displaying the
209 captured logs and errors. First let's redisplay the results enumerated by adding
214 ~/frr/tests/topotests# ./analyze.py -Ar run-save -E
215 0 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge
216 1 ospf_basic_functionality/test_ospf_lan.py::test_ospf_lan_tc1_p0
217 2 bgp_gr_functionality_topo2/test_bgp_gr_functionality_topo2.py::test_BGP_GR_10_p2
218 3 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_routingTable
220 Now to look at the error message for a failed test we use ``-T N`` where N is
221 the number of the test we are interested in along with ``--errmsg`` option.
225 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0 --errmsg
226 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge: AssertionError: BGP did not converge:
228 IPv4 Unicast Summary (VIEW 1):
229 BGP router identifier 172.30.1.1, local AS number 100 vrf-id -1
231 RIB entries 1, using 184 bytes of memory
232 Peers 3, using 2169 KiB of memory
234 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
235 172.16.1.1 4 65001 0 0 0 0 0 never Connect 0 N/A
236 172.16.1.2 4 65002 0 0 0 0 0 never Connect 0 N/A
237 172.16.1.5 4 65005 0 0 0 0 0 never Connect 0 N/A
239 Total number of neighbors 3
243 Now to look at the full text of the error for a failed test we use ``-T N``
244 where N is the number of the test we are interested in along with ``--errtext``
249 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0 --errtext
250 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge: def test_bgp_converge():
251 "Check for BGP converged on all peers and BGP views"
257 # Bail out with error if a router fails to converge
258 bgpStatus = net["r%s" % i].cmd('vtysh -c "show ip bgp view %s summary"' % view)
259 > assert False, "BGP did not converge:\n%s" % bgpStatus
260 E AssertionError: BGP did not converge:
262 E IPv4 Unicast Summary (VIEW 1):
263 E BGP router identifier 172.30.1.1, local AS number 100 vrf-id -1
265 E Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
266 E 172.16.1.1 4 65001 0 0 0 0 0 never Connect 0 N/A
267 E 172.16.1.2 4 65002 0 0 0 0 0 never Connect 0 N/A
270 To look at the full capture for a test including the stdout and stderr which
271 includes full debug logs, just use the ``-T N`` option without the ``--errmsg``
272 or ``--errtext`` options.
276 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0
277 @classname: bgp_multiview_topo1.test_bgp_multiview_topo1
278 @name: test_bgp_converge
280 @message: AssertionError: BGP did not converge:
282 system-out: --------------------------------- Captured Log ---------------------------------
283 2021-08-09 02:55:06,581 DEBUG: lib.micronet_compat.topo: Topo(unnamed): Creating
284 2021-08-09 02:55:06,581 DEBUG: lib.micronet_compat.topo: Topo(unnamed): addHost r1
286 2021-08-09 02:57:16,932 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary" 2> /dev/null | grep ^[0-9] | grep -vP " 11\\s+(\\d+)"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
287 2021-08-09 02:57:22,290 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary" 2> /dev/null | grep ^[0-9] | grep -vP " 11\\s+(\\d+)"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
288 2021-08-09 02:57:27,636 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
289 --------------------------------- Captured Out ---------------------------------
290 system-err: --------------------------------- Captured Err ---------------------------------
299 sudo -E pytest ./test_to_be_run.py
301 For example, and assuming you are inside the frr directory:
305 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
306 sudo -E pytest ./test_bgp_l3vpn_to_bgp_vrf.py
308 For further options, refer to pytest documentation.
310 Test will set exit code which can be used with ``git bisect``.
312 For the simulated topology, see the description in the python file.
314 StdErr log from daemos after exit
315 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
317 To enable the reporting of any messages seen on StdErr after the daemons exit,
318 the following env variable can be set::
320 export TOPOTESTS_CHECK_STDERR=Yes
322 (The value doesn't matter at this time. The check is whether the env
323 variable exists or not.) There is no pass/fail on this reporting; the
324 Output will be reported to the console.
326 Collect Memory Leak Information
327 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
329 FRR processes can report unfreed memory allocations upon exit. To
330 enable the reporting of memory leaks, define an environment variable
331 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
333 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
335 This will enable the check and output to console and the writing of
336 the information to files with the given prefix (followed by testname),
337 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
340 Running Topotests with AddressSanitizer
341 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
343 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
344 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
345 information on AddressSanitizer, see
346 https://github.com/google/sanitizers/wiki/AddressSanitizer.
348 The checks are done automatically in the library call of ``checkRouterRunning``
349 (ie at beginning of tests when there is a check for all daemons running). No
350 changes or extra configuration for topotests is required beside compiling the
351 suite with AddressSanitizer enabled.
353 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
354 If found, then this is added with context (calling test) to
355 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
357 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
358 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
359 for ``master`` branch:
363 git clone https://github.com/FRRouting/frr.git
367 --enable-address-sanitizer \
368 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
369 --localstatedir=/var/run/frr \
370 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
371 --with-moduledir=/usr/lib/frr/modules \
372 --enable-multipath=0 --enable-rtadv \
373 --enable-tcp-zebra --enable-fpm --enable-pimd \
377 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
378 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
380 and create ``frr`` user and ``frrvty`` group as shown above.
382 Debugging Topotest Failures
383 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
385 Install and run tests inside ``tmux`` or ``byobu`` for best results.
387 ``XTerm`` is also fully supported. GNU ``screen`` can be used in most
388 situations; however, it does not work as well with launching ``vtysh`` or shell
391 For the below debugging options which launch programs or CLIs, topotest should
392 be run within ``tmux`` (or ``screen``)_, as ``gdb``, the shell or ``vtysh`` will
393 be launched using that windowing program, otherwise ``xterm`` will be attempted
394 to launch the given programs.
396 NOTE: you must run the topotest (pytest) such that your DISPLAY, STY or TMUX
397 environment variables are carried over. You can do this by passing the
398 ``-E`` flag to ``sudo`` or you can modify your ``/etc/sudoers`` config to
399 automatically pass that environment variable through to the ``sudo``
402 .. _screen: https://www.gnu.org/software/screen/
403 .. _tmux: https://github.com/tmux/tmux/wiki
408 One can view and capture packets on any of the networks or interfaces defined by
409 the topotest by specifying the ``--pcap=NET|INTF|all[,NET|INTF,...]`` CLI option
410 as shown in the examples below.
414 # Capture on all networks in isis_topo1 test
415 sudo -E pytest isis_topo1 --pcap=all
417 # Capture on `sw1` network
418 sudo -E pytest isis_topo1 --pcap=sw1
420 # Capture on `sw1` network and on interface `eth0` on router `r2`
421 sudo -E pytest isis_topo1 --pcap=sw1,r2:r2-eth0
423 For each capture a window is opened displaying a live summary of the captured
424 packets. Additionally, the entire packet stream is captured in a pcap file in
425 the tests log directory e.g.,::
429 $ sudo -E pytest isis_topo1 --pcap=sw1,r2:r2-eth0
431 $ ls -l /tmp/topotests/isis_topo1.test_isis_topo1/
432 -rw------- 1 root root 45172 Apr 19 05:30 capture-r2-r2-eth0.pcap
433 -rw------- 1 root root 48412 Apr 19 05:30 capture-sw1.pcap
436 Viewing Live Daemon Logs
437 """"""""""""""""""""""""
439 One can live view daemon or the frr logs in separate windows using the
440 ``--logd`` CLI option as shown below.
444 # View `ripd` logs on all routers in test
445 sudo -E pytest rip_allow_ecmp --logd=ripd
447 # View `ripd` logs on all routers and `mgmtd` log on `r1`
448 sudo -E pytest rip_allow_ecmp --logd=ripd --logd=mgmtd,r1
450 For each capture a window is opened displaying a live summary of the captured
451 packets. Additionally, the entire packet stream is captured in a pcap file in
452 the tests log directory e.g.,::
454 When using a unified log file `frr.log` one substitutes `frr` for the daemon
455 name in the ``--logd`` CLI option, e.g.,
459 # View `frr` log on all routers in test
460 sudo -E pytest some_test_suite --logd=frr
462 Spawning Debugging CLI, ``vtysh`` or Shells on Routers on Test Failure
463 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
465 One can have a debugging CLI invoked on test failures by specifying the
466 ``--cli-on-error`` CLI option as shown in the example below.
470 sudo -E pytest --cli-on-error all-protocol-startup
472 The debugging CLI can run shell or vtysh commands on any combination of routers
473 It can also open shells or vtysh in their own windows for any combination of
474 routers. This is usually the most useful option when debugging failures. Here is
475 the help command from within a CLI launched on error:
479 test_bgp_multiview_topo1/test_bgp_routingTable> help
482 cli :: open a secondary CLI window
487 HOST can be a host or one of the following:
489 - '.' for the parent munet
490 - a regex specified between '/' (e.g., '/rtr.*/')
493 logd HOST [HOST ...] DAEMON :: tail -f on the logfile of the given DAEMON for the given HOST[S]
494 pcap NETWORK :: capture packets from NETWORK into file capture-NETWORK.pcap the command is run within a new window which also shows packet summaries. NETWORK can also be an interface specified as HOST:INTF. To capture inside the host namespace.
495 stderr HOST [HOST ...] DAEMON :: tail -f on the stderr of the given DAEMON for the given HOST[S]
496 stdlog HOST [HOST ...] :: tail -f on the `frr.log` for the given HOST[S]
497 stdout HOST [HOST ...] DAEMON :: tail -f on the stdout of the given DAEMON for the given HOST[S]
498 term HOST [HOST ...] :: open terminal[s] (TMUX or XTerm) on HOST[S], * for all
499 vtysh ROUTER [ROUTER ...] ::
500 xterm HOST [HOST ...] :: open XTerm[s] on HOST[S], * for all
502 [ROUTER ...] COMMAND :: execute vtysh COMMAND on the router[s]
503 [HOST ...] sh <SHELL-COMMAND> :: execute <SHELL-COMMAND> on hosts
504 [HOST ...] shi <INTERACTIVE-COMMAND> :: execute <INTERACTIVE-COMMAND> on HOST[s]
506 test_bgp_multiview_topo1/test_bgp_routingTable> r1 show int br
507 ------ Host: r1 ------
508 Interface Status VRF Addresses
509 --------- ------ --- ---------
514 r1-eth0 up default 172.16.1.254/24
515 r1-stub up default 172.20.0.1/28
517 ----------------------
518 test_bgp_multiview_topo1/test_bgp_routingTable>
520 Additionally, one can have ``vtysh`` or a shell launched on all routers when a
521 test fails. To launch the given process on each router after a test failure
522 specify one of ``--shell-on-error`` or ``--vtysh-on-error``.
524 Spawning ``vtysh`` or Shells on Routers
525 """""""""""""""""""""""""""""""""""""""
527 Topotest can automatically launch a shell or ``vtysh`` for any or all routers in
528 a test. This is enabled by specifying 1 of 2 CLI arguments ``--shell`` or
529 ``--vtysh``. Both of these options can be set to a single router value, multiple
530 comma-seperated values, or ``all``.
532 When either of these options are specified topotest will pause after setup and
533 each test to allow for inspection of the router state.
535 Here's an example of launching ``vtysh`` on routers ``rt1`` and ``rt2``.
539 sudo -E pytest --vtysh=rt1,rt2 all-protocol-startup
544 Topotest can automatically launch any daemon with ``gdb``, possibly setting
545 breakpoints for any test run. This is enabled by specifying 1 or 2 CLI arguments
546 ``--gdb-routers`` and ``--gdb-daemons``. Additionally ``--gdb-breakpoints`` can
547 be used to automatically set breakpoints in the launched ``gdb`` processes.
549 Each of these options can be set to a single value, multiple comma-seperated
550 values, or ``all``. If ``--gdb-routers`` is empty but ``--gdb_daemons`` is set
551 then the given daemons will be launched in ``gdb`` on all routers in the test.
552 Likewise if ``--gdb_routers`` is set, but ``--gdb_daemons`` is empty then all
553 daemons on the given routers will be launched in ``gdb``.
555 Here's an example of launching ``zebra`` and ``bgpd`` inside ``gdb`` on router
556 ``r1`` with a breakpoint set on ``nb_config_diff``
560 sudo -E pytest --gdb-routers=r1 \
561 --gdb-daemons=bgpd,zebra \
562 --gdb-breakpoints=nb_config_diff \
565 Detecting Memleaks with Valgrind
566 """"""""""""""""""""""""""""""""
568 Topotest can automatically launch all daemons with ``valgrind`` to check for
569 memleaks. This is enabled by specifying 1 or 2 CLI arguments.
570 ``--valgrind-memleaks`` will enable general memleak detection, and
571 ``--valgrind-extra`` enables extra functionality including generating a
572 suppression file. The suppression file ``tools/valgrind.supp`` is used when
573 memleak detection is enabled.
577 sudo -E pytest --valgrind-memleaks all-protocol-startup
579 Collecting Performance Data using perf(1)
580 """""""""""""""""""""""""""""""""""""""""
582 Topotest can automatically launch any daemon under ``perf(1)`` to collect
583 performance data. The daemon is run in non-daemon mode with ``perf record -g``.
584 The ``perf.data`` file will be saved in the router specific directory under the
587 Here's an example of collecting performance data from ``mgmtd`` on router ``r1``
588 during the config_timing test.
592 $ sudo -E pytest --perf=mgmtd,r1 config_timing
594 $ find /tmp/topotests/ -name '*perf.data*'
595 /tmp/topotests/config_timing.test_config_timing/r1/perf.data
597 To specify different arguments for ``perf record``, one can use the
598 ``--perf-options`` this will replace the ``-g`` used by default.
600 .. _topotests_docker:
602 Running Tests with Docker
603 -------------------------
605 There is a Docker image which allows to run topotests.
610 If you have Docker installed, you can run the topotests in Docker. The easiest
611 way to do this, is to use the make targets from this repository.
613 Your current user needs to have access to the Docker daemon. Alternatively you
614 can run these commands as root.
620 This command will pull the most recent topotests image from Dockerhub, compile
621 FRR inside of it, and run the topotests.
626 Internally, the topotests make target uses a shell script to pull the image and
627 spawn the Docker container.
629 There are several environment variables which can be used to modify the
630 behavior of the script, these can be listed by calling it with ``-h``:
634 ./tests/topotests/docker/frr-topotests.sh -h
636 For example, a volume is used to cache build artifacts between multiple runs of
637 the image. If you need to force a complete recompile, you can set
642 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
644 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
645 arguments and the first one starts with ``/`` or ``./``, they will replace the
646 call to pytest. If the appended arguments do not match this patttern, they will
647 be provided to pytest as arguments. So, to run a specific test with more
652 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
654 And to compile FRR but drop into a shell instead of running pytest:
658 ./tests/topotests/docker/frr-topotests.sh /bin/bash
663 The Docker image just includes all the components to run the topotests, but not
664 the topotests themselves. So if you just want to write tests and don't want to
665 make changes to the environment provided by the Docker image. You don't need to
666 build your own Docker image if you do not want to.
668 When developing new tests, there is one caveat though: The startup script of
669 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
670 pollution of the container with build artefacts from the host. This will also
671 result in your newly written tests being unavailable in the container unless at
672 least added to the index with ``git-add``.
674 If you do want to test changes to the Docker image, you can locally build the
675 image and run the tests without pulling from the registry using the following
681 TOPOTEST_PULL=0 make topotests
684 .. _topotests-guidelines:
692 To run the whole suite of tests the following commands must be executed at the
693 top level directory of topotest:
697 $ # Change to the top level directory of topotests.
698 $ cd path/to/topotests
699 $ # Tests must be run as root, since micronet requires it.
702 In order to run a specific test, you can use the following command:
706 $ # running a specific topology
707 $ sudo -E pytest ospf-topo1/
708 $ # or inside the test folder
710 $ sudo -E pytest # to run all tests inside the directory
711 $ sudo -E pytest test_ospf_topo1.py # to run a specific test
712 $ # or outside the test folder
714 $ sudo -E pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
716 The output of the tested daemons will be available at the temporary folder of
721 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
723 zebra.err # zebra stderr output
724 zebra.log # zebra log file
725 zebra.out # zebra stdout output
728 You can also run memory leak tests to get reports:
732 $ # Set the environment variable to apply to a specific test...
733 $ sudo -E env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
734 $ # ...or apply to all tests adding this line to the configuration file
735 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
736 $ # You can also use your editor
738 $ # After running tests you should see your files:
739 $ ls /tmp/memleak_report_*
740 memleak_report_test_ospf_topo1.txt
745 This section will guide you in all recommended steps to produce a standard
748 This is the recommended test writing routine:
750 - Write a topology (Graphviz recommended)
751 - Obtain configuration files
752 - Write the test itself
753 - Format the new code using `black <https://github.com/psf/black>`_
754 - Create a Pull Request
756 Some things to keep in mind:
758 - BGP tests MUST use generous convergence timeouts - you must ensure
759 that any test involving BGP uses a convergence timeout of at least
761 - Topotests are run on a range of Linux versions: if your test
762 requires some OS-specific capability (like mpls support, or vrf
763 support), there are test functions available in the libraries that
764 will help you determine whether your test should run or be skipped.
765 - Avoid including unstable data in your test: don't rely on link-local
766 addresses or ifindex values, for example, because these can change
768 - Using sleep is almost never appropriate. As an example: if the test resets the
769 peers in BGP, the test should look for the peers re-converging instead of just
770 sleeping an arbitrary amount of time and continuing on. See
771 ``verify_bgp_convergence`` as a good example of this. In particular look at
772 it's use of the ``@retry`` decorator. If you are having troubles figuring out
773 what to look for, please do not be afraid to ask.
774 - Don't duplicate effort. There exists many protocol utility functions that can
775 be found in their eponymous module under ``tests/topotests/lib/`` (e.g.,
780 Topotest File Hierarchy
781 """""""""""""""""""""""
783 Before starting to write any tests one must know the file hierarchy. The
784 repository hierarchy looks like this:
788 $ cd path/to/topotest
791 ./README.md # repository read me
792 ./GUIDELINES.md # this file
793 ./conftest.py # test hooks - pytest related functions
794 ./example-test # example test folder
795 ./example-test/__init__.py # python package marker - must always exist.
796 ./example-test/test_template.jpg # generated topology picture - see next section
797 ./example-test/test_template.dot # Graphviz dot file
798 ./example-test/test_template.py # the topology plus the test
800 ./ospf-topo1 # the ospf topology test
801 ./ospf-topo1/r1 # router 1 configuration files
802 ./ospf-topo1/r1/zebra.conf # zebra configuration file
803 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
804 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
805 # removed other for shortness sake
807 ./lib # shared test/topology functions
808 ./lib/topogen.py # topogen implementation
809 ./lib/topotest.py # topotest implementation
811 Guidelines for creating/editing topotest:
813 - New topologies that don't fit the existing directories should create its own
814 - Always remember to add the ``__init__.py`` to new folders, this makes auto
815 complete engines and pylint happy
816 - Router (Quagga/FRR) specific code should go on topotest.py
817 - Generic/repeated router actions should have an abstraction in
819 - Generic/repeated non-router code should go to topotest.py
820 - pytest related code should go to conftest.py (e.g. specialized asserts)
822 Defining the Topology
823 """""""""""""""""""""
825 The first step to write a new test is to define the topology. This step can be
826 done in many ways, but the recommended is to use Graphviz to generate a drawing
827 of the topology. It allows us to see the topology graphically and to see the
828 names of equipment, links and addresses.
830 Here is an example of Graphviz dot file that generates the template topology
831 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
832 get outdated, please see the linked file)::
854 label="s1\n192.168.0.0/24",
860 label="s2\n192.168.1.0/24",
866 r1 -- s1 [label="eth0\n.1"];
868 r1 -- s2 [label="eth1\n.100"];
869 r2 -- s2 [label="eth0\n.1"];
872 Here is the produced graph:
896 label="s1\n192.168.0.0/24",
902 label="s2\n192.168.1.0/24",
908 r1 -- s1 [label="eth0\n.1"];
910 r1 -- s2 [label="eth1\n.100"];
911 r2 -- s2 [label="eth0\n.1"];
914 Generating / Obtaining Configuration Files
915 """"""""""""""""""""""""""""""""""""""""""
917 In order to get the configuration files or command output for each router, we
918 need to run the topology and execute commands in ``vtysh``. The quickest way to
919 achieve that is writing the topology building code and running the topology.
921 To bootstrap your test topology, do the following steps:
923 - Copy the template test
928 $ touch new-topo/__init__.py
929 $ cp example-test/test_template.py new-topo/test_new_topo.py
931 - Modify the template according to your dot file
933 Here is the template topology described in the previous section in python code:
942 If more specialized topology definitions, or router initialization arguments are
943 required a build function can be used instead of a dictionary:
947 def build_topo(tgen):
951 for routern in range(1, 3):
952 tgen.add_router("r{}".format(routern))
954 # Create a switch with just one router connected to it to simulate a
956 switch = tgen.add_switch("s1")
957 switch.add_link(tgen.gears["r1"])
959 # Create a connection between r1 and r2
960 switch = tgen.add_switch("s2")
961 switch.add_link(tgen.gears["r1"])
962 switch.add_link(tgen.gears["r2"])
966 Topogen allows us to run the topology without running any tests, you can do
967 that using the following example commands:
971 $ # Running your bootstraped topology
972 $ sudo -E pytest -s --topology-only new-topo/test_new_topo.py
973 $ # Running the test_template.py topology
974 $ sudo -E pytest -s --topology-only example-test/test_template.py
975 $ # Running the ospf_topo1.py topology
976 $ sudo -E pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
978 Parameters explanation:
984 Actives input/output capture. If this is not specified a new window will be
985 opened for the interactive CLI, otherwise it will be activated inline.
987 .. option:: --topology-only
989 Don't run any tests, just build the topology.
991 After executing the commands above, you should get the following terminal
996 frr/tests/topotests# sudo -E pytest -s --topology-only ospf_topo1/test_ospf_topo1.py
997 ============================= test session starts ==============================
998 platform linux -- Python 3.9.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
999 rootdir: /home/chopps/w/frr/tests/topotests, configfile: pytest.ini
1000 plugins: forked-1.3.0, xdist-2.3.0
1006 The last line shows us that we are now using the CLI (Command Line
1007 Interface), from here you can call your router ``vtysh`` or even bash.
1009 Here's the help text:
1017 sh [hosts] <shell-command> :: execute <shell-command> on <host>
1018 term [hosts] :: open shell terminals for hosts
1019 vtysh [hosts] :: open vtysh terminals for hosts
1020 [hosts] <vtysh-command> :: execute vtysh-command on hosts
1022 Here are some commands example:
1026 unet> sh r1 ping 10.0.3.1
1027 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
1028 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
1029 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
1030 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
1032 --- 10.0.3.1 ping statistics ---
1033 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
1034 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
1037 Building configuration...
1039 Current configuration:
1041 frr version 8.1-dev-my-manual-build
1042 frr defaults traditional
1044 log file /tmp/topotests/ospf_topo1.test_ospf_topo1/r1/zebra.log
1049 ------ Host: r1 ------
1050 zebra ospfd ospf6d staticd
1051 ------- End: r1 ------
1052 ------ Host: r2 ------
1053 zebra ospfd ospf6d staticd
1054 ------- End: r2 ------
1055 ------ Host: r3 ------
1056 zebra ospfd ospf6d staticd
1057 ------- End: r3 ------
1058 ------ Host: r4 ------
1059 zebra ospfd ospf6d staticd
1060 ------- End: r4 ------
1062 After you successfully configured your topology, you can obtain the
1063 configuration files (per-daemon) using the following commands:
1067 unet> sh r3 vtysh -d ospfd
1069 Hello, this is FRRouting (version 3.1-devrzalamena-build).
1070 Copyright 1996-2005 Kunihiro Ishiguro, et al.
1072 r1# show running-config
1073 Building configuration...
1075 Current configuration:
1077 frr version 3.1-devrzalamena-build
1078 frr defaults traditional
1079 no service integrated-vtysh-config
1084 ospf router-id 10.0.255.3
1086 redistribute connected
1088 network 10.0.3.0/24 area 0
1089 network 10.0.10.0/24 area 0
1090 network 172.16.0.0/24 area 1
1097 You can also login to the node specified by nsenter using bash, etc.
1098 A pid file for each node will be created in the relevant test dir.
1099 You can run scripts inside the node, or use vtysh's <tab> or <?> feature.
1104 # cd tests/topotests/srv6_locator
1105 # ./test_srv6_locator.py --topology-only
1106 unet> r1 show segment-routing srv6 locator
1108 Name ID Prefix Status
1109 -------------------- ------- ------------------------ -------
1110 loc1 1 2001:db8:1:1::/64 Up
1111 loc2 2 2001:db8:2:2::/64 Up
1114 # nsenter -a -t $(cat /tmp/topotests/srv6_locator.test_srv6_locator/r1.pid) bash --norc
1116 r1# r1 show segment-routing srv6 locator
1118 Name ID Prefix Status
1119 -------------------- ------- ------------------------ -------
1120 loc1 1 2001:db8:1:1::/64 Up
1121 loc2 2 2001:db8:2:2::/64 Up
1126 Test topologies should always be bootstrapped from
1127 :file:`tests/topotests/example_test/test_template.py` because it contains
1128 important boilerplate code that can't be avoided, like:
1134 # For all routers arrange for:
1135 # - starting zebra using config file from <rtrname>/zebra.conf
1136 # - starting ospfd using an empty config file.
1137 for rname, router in router_list.items():
1138 router.load_config(TopoRouter.RD_ZEBRA, "zebra.conf")
1139 router.load_config(TopoRouter.RD_OSPF)
1142 - The topology definition or build function
1151 def build_topo(tgen):
1152 # topology build code
1155 - pytest setup/teardown fixture to start the topology and supply ``tgen``
1161 @pytest.fixture(scope="module")
1163 "Setup/Teardown the environment and provide tgen argument to tests"
1165 tgen = Topogen(topodef, module.__name__)
1167 tgen = Topogen(build_topo, module.__name__)
1171 # Start and configure the router daemons
1174 # Provide tgen as argument to each test function
1177 # Teardown after last test runs
1178 tgen.stop_topology()
1183 - Directory name for a new topotest must not contain hyphen (``-``) characters.
1184 To separate words, use underscores (``_``). For example, ``tests/topotests/bgp_new_example``.
1185 - Test code should always be declared inside functions that begin with the
1186 ``test_`` prefix. Functions beginning with different prefixes will not be run
1188 - Configuration files and long output commands should go into separated files
1189 inside folders named after the equipment.
1190 - Tests must be able to run without any interaction. To make sure your test
1191 conforms with this, run it without the :option:`-s` parameter.
1192 - Use `black <https://github.com/psf/black>`_ code formatter before creating
1193 a pull request. This ensures we have a unified code style.
1194 - Mark test modules with pytest markers depending on the daemons used during the
1195 tests (see :ref:`topotests-markers`)
1196 - Always use IPv4 :rfc:`5737` (``192.0.2.0/24``, ``198.51.100.0/24``,
1197 ``203.0.113.0/24``) and IPv6 :rfc:`3849` (``2001:db8::/32``) ranges reserved
1202 - Keep results in stack variables, so people inspecting code with ``pdb`` can
1203 easily print their values.
1209 assert foobar(router1, router2)
1215 result = foobar(router1, router2)
1218 - Use ``assert`` messages to indicate where the test failed.
1224 for router in router_list:
1226 assert condition, 'Router "{}" condition failed'.format(router.name)
1231 The most effective ways to inspect topology tests are:
1233 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
1234 appear when an assertion fails
1236 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
1238 - Set a breakpoint in the test code with ``pdb``
1244 # Add the pdb import at the beginning of the file
1248 # Add a breakpoint where you think the problem is
1254 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
1255 shell allows us to run many useful operations like:
1257 - Setting breaking point on file/function/conditions (e.g. ``break``,
1259 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
1260 - Running python code
1264 The TopoGear (equipment abstraction class) implements the ``__str__`` method
1265 that allows the user to inspect equipment information.
1267 Example of pdb usage:
1271 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
1272 -> for rnum in range(1, 5):
1274 Documented commands (type help <topic>):
1275 ========================================
1276 EOF bt cont enable jump pp run unt
1277 a c continue exit l q s until
1278 alias cl d h list quit step up
1279 args clear debug help n r tbreak w
1280 b commands disable ignore next restart u whatis
1281 break condition down j p return unalias where
1283 Miscellaneous help topics:
1284 ==========================
1287 Undocumented commands:
1288 ======================
1292 116 title2="Expected output")
1294 118 def test_ospf_convergence():
1295 119 "Test OSPF daemon convergence"
1297 121 -> for rnum in range(1, 5):
1298 122 router = 'r{}'.format(rnum)
1300 124 # Load expected results from the command
1301 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1302 126 expected = open(reffile).read()
1304 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
1305 -> router = 'r{}'.format(rnum)
1307 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
1308 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1313 (Pdb) tgen = get_topogen()
1314 (Pdb) pp tgen.gears[router]
1315 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
1316 (Pdb) pp str(tgen.gears[router])
1317 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
1320 121 for rnum in range(1, 5):
1321 122 router = 'r{}'.format(rnum)
1323 124 # Load expected results from the command
1324 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1325 126 expected = open(reffile).read()
1327 128 # Run test function until we get an result. Wait at most 60 seconds.
1328 129 test_func = partial(compare_show_ip_ospf, router, expected)
1329 130 result, diff = topotest.run_and_expect(test_func, '',
1330 (Pdb) router1 = tgen.gears[router]
1331 (Pdb) router1.vtysh_cmd('show ip ospf route')
1332 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
1336 To enable more debug messages in other Topogen subsystems, more
1337 logging messages can be displayed by modifying the test configuration file
1343 # Change the default verbosity line from 'info'...
1348 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
1349 To learn/remember common code snippets see :ref:`topotests-snippets`.
1351 Before creating a new topology, make sure that there isn't one already that
1352 does what you need. If nothing is similar, then you may create a new topology,
1353 preferably, using the newest template
1354 (:file:`tests/topotests/example-test/test_template.py`).
1356 .. include:: topotests-markers.rst
1358 .. include:: topotests-snippets.rst
1363 All the configs and scripts are licensed under a ISC-style license. See Python
1364 scripts for details.