6 Topotests is a suite of topology tests for FRR built on top of micronet.
11 Topotests run under python3. Additionally, for ExaBGP (which is used
12 in some of the BGP tests) an older python2 version (and the python2
13 version of ``pip``) must be installed.
15 Tested with Ubuntu 20.04,Ubuntu 18.04, and Debian 11.
17 Instructions are the same for all setups (i.e. ExaBGP is only used for
20 Tshark is only required if you enable any packet captures on test runs.
22 Valgrind is only required if you enable valgrind on test runs.
24 Installing Topotest Requirements
25 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
37 python3 -m pip install wheel
38 python3 -m pip install 'pytest>=6.2.4'
39 python3 -m pip install 'pytest-xdist>=2.3.0'
40 python3 -m pip install 'scapy>=2.4.5'
41 python3 -m pip install xmltodict
42 # Use python2 pip to install older ExaBGP
43 python2 -m pip install 'exabgp<4.0.0'
44 useradd -d /var/run/exabgp/ -s /bin/false exabgp
46 # To enable the gRPC topotest install:
47 python3 -m pip install grpcio grpcio-tools
49 # Install Socat tool to run PIMv6 tests,
50 # Socat code can be taken from below url,
51 # which has latest changes done for PIMv6,
53 https://github.com/opensourcerouting/socat/
59 Optional, will give better output.
63 disable apport (which move core files)
65 Set ``enabled=0`` in ``/etc/default/apport``.
67 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
69 #<domain> <type> <item> <value>
71 root soft core unlimited
73 root hard core unlimited
75 Reboot for options to take effect.
77 SNMP Utilities Installation
78 """""""""""""""""""""""""""
80 To run SNMP test you need to install SNMP utilities and MIBs. Unfortunately
81 there are some errors in the upstream MIBS which need to be patched up. The
82 following steps will get you there on Ubuntu 20.04.
86 apt install libsnmp-dev
87 apt install snmpd snmp
88 apt install snmp-mibs-downloader
90 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/iana/IANA-IPPM-METRICS-REGISTRY-MIB -O /usr/share/snmp/mibs/iana/IANA-IPPM-METRICS-REGISTRY-MIB
91 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/ietf/SNMPv2-PDU -O /usr/share/snmp/mibs/ietf/SNMPv2-PDU
92 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/ietf/IPATM-IPMC-MIB -O /usr/share/snmp/mibs/ietf/IPATM-IPMC-MIB
93 edit /etc/snmp/snmp.conf to look like this
94 # As the snmp packages come without MIB files due to license reasons, loading
95 # of MIBs is disabled by default. If you added the MIBs you can reenable
96 # loading them by commenting out the following line.
103 FRR needs to be installed separately. It is assume to be configured like the
104 standard Ubuntu Packages:
106 - Binaries in :file:`/usr/lib/frr`
107 - State Directory :file:`/var/run/frr`
108 - Running under user ``frr``, group ``frr``
109 - vtygroup: ``frrvty``
110 - config directory: :file:`/etc/frr`
111 - For FRR Packages, install the dbg package as well for coredump decoding
113 No FRR config needs to be done and no FRR daemons should be run ahead of the
114 test. They are all started as part of the test.
119 If you prefer to manually build FRR, then use the following suggested config:
125 --localstatedir=/var/run/frr \
126 --sbindir=/usr/lib/frr \
127 --sysconfdir=/etc/frr \
132 --enable-multipath=64 \
135 --enable-vty-group=frrvty \
136 --enable-snmp=agentx \
137 --with-pkg-extra-version=-my-manual-build
139 And create ``frr`` user and ``frrvty`` group as follows:
143 addgroup --system --gid 92 frr
144 addgroup --system --gid 85 frrvty
145 adduser --system --ingroup frr --home /var/run/frr/ \
146 --gecos "FRRouting suite" --shell /bin/false frr
147 usermod -G frrvty frr
152 Configure your sudo environment
153 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
155 Topotests must be run as root. Normally this will be accomplished through the
156 use of the ``sudo`` command. In order for topotests to be able to open new
157 windows (either XTerm or byobu/screen/tmux windows) certain environment
158 variables must be passed through the sudo command. One way to do this is to
159 specify the ``-E`` flag to ``sudo``. This will carry over most if not all
160 your environment variables include ``PATH``. For example:
164 sudo -E python3 -m pytest -s -v
166 If you do not wish to use ``-E`` (e.g., to avoid ``sudo`` inheriting
167 ``PATH``) you can modify your `/etc/sudoers` config file to specifically pass
168 the environment variables required by topotests. Add the following commands to
169 your ``/etc/sudoers`` config file.
173 Defaults env_keep="TMUX"
174 Defaults env_keep+="TMUX_PANE"
175 Defaults env_keep+="STY"
176 Defaults env_keep+="DISPLAY"
178 If there was already an ``env_keep`` configuration there be sure to use the
179 ``+=`` rather than ``=`` on the first line above as well.
182 Execute all tests in distributed test mode
183 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
187 sudo -E pytest -s -v -nauto --dist=loadfile
189 The above command must be executed from inside the topotests directory.
191 All test\_\* scripts in subdirectories are detected and executed (unless
192 disabled in ``pytest.ini`` file). Pytest will execute up to N tests in parallel
193 where N is based on the number of cores on the host.
195 Analyze Test Results (``analyze.py``)
196 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
198 By default router and execution logs are saved in ``/tmp/topotests`` and an XML
199 results file is saved in ``/tmp/topotests/topotests.xml``. An analysis tool
200 ``analyze.py`` is provided to archive and analyze these results after the run
203 After the test run completes one should pick an archive directory to store the
204 results in and pass this value to ``analyze.py``. On first execution the results
205 are moved to that directory from ``/tmp/topotests``. Subsequent runs of
206 ``analyze.py`` with the same args will use that directories contents for instead
207 of copying any new results from ``/tmp``. Below is an example of this which also
208 shows the default behavior which is to display all failed and errored tests in
213 ~/frr/tests/topotests# ./analyze.py -Ar run-save
214 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge
215 ospf_basic_functionality/test_ospf_lan.py::test_ospf_lan_tc1_p0
216 bgp_gr_functionality_topo2/test_bgp_gr_functionality_topo2.py::test_BGP_GR_10_p2
217 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_routingTable
219 Here we see that 4 tests have failed. We can dig deeper by displaying the
220 captured logs and errors. First let's redisplay the results enumerated by adding
225 ~/frr/tests/topotests# ./analyze.py -Ar run-save -E
226 0 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge
227 1 ospf_basic_functionality/test_ospf_lan.py::test_ospf_lan_tc1_p0
228 2 bgp_gr_functionality_topo2/test_bgp_gr_functionality_topo2.py::test_BGP_GR_10_p2
229 3 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_routingTable
231 Now to look at the error message for a failed test we use ``-T N`` where N is
232 the number of the test we are interested in along with ``--errmsg`` option.
236 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0 --errmsg
237 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge: AssertionError: BGP did not converge:
239 IPv4 Unicast Summary (VIEW 1):
240 BGP router identifier 172.30.1.1, local AS number 100 vrf-id -1
242 RIB entries 1, using 184 bytes of memory
243 Peers 3, using 2169 KiB of memory
245 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
246 172.16.1.1 4 65001 0 0 0 0 0 never Connect 0 N/A
247 172.16.1.2 4 65002 0 0 0 0 0 never Connect 0 N/A
248 172.16.1.5 4 65005 0 0 0 0 0 never Connect 0 N/A
250 Total number of neighbors 3
254 Now to look at the error text for a failed test we can use ``-T RANGES`` where
255 ``RANGES`` can be a number (e.g., ``5``), a range (e.g., ``0-10``), or a comma
256 separated list numbers and ranges (e.g., ``5,10-20,30``) of the test cases we
257 are interested in along with ``--errtext`` option. In the example below we'll
258 select the first failed test case.
262 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0 --errtext
263 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge: def test_bgp_converge():
264 "Check for BGP converged on all peers and BGP views"
270 # Bail out with error if a router fails to converge
271 bgpStatus = net["r%s" % i].cmd('vtysh -c "show ip bgp view %s summary"' % view)
272 > assert False, "BGP did not converge:\n%s" % bgpStatus
273 E AssertionError: BGP did not converge:
275 E IPv4 Unicast Summary (VIEW 1):
276 E BGP router identifier 172.30.1.1, local AS number 100 vrf-id -1
278 E Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
279 E 172.16.1.1 4 65001 0 0 0 0 0 never Connect 0 N/A
280 E 172.16.1.2 4 65002 0 0 0 0 0 never Connect 0 N/A
283 To look at the full capture for a test including the stdout and stderr which
284 includes full debug logs, use ``--full`` option, or specify a ``-T RANGES`` without
285 specifying ``--errmsg`` or ``--errtext``.
289 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0
290 @classname: bgp_multiview_topo1.test_bgp_multiview_topo1
291 @name: test_bgp_converge
293 @message: AssertionError: BGP did not converge:
295 system-out: --------------------------------- Captured Log ---------------------------------
296 2021-08-09 02:55:06,581 DEBUG: lib.micronet_compat.topo: Topo(unnamed): Creating
297 2021-08-09 02:55:06,581 DEBUG: lib.micronet_compat.topo: Topo(unnamed): addHost r1
299 2021-08-09 02:57:16,932 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary" 2> /dev/null | grep ^[0-9] | grep -vP " 11\\s+(\\d+)"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
300 2021-08-09 02:57:22,290 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary" 2> /dev/null | grep ^[0-9] | grep -vP " 11\\s+(\\d+)"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
301 2021-08-09 02:57:27,636 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
302 --------------------------------- Captured Out ---------------------------------
303 system-err: --------------------------------- Captured Err ---------------------------------
308 There are 4 types of test results, [e]rrored, [f]ailed, [p]assed, and
309 [s]kipped. One can select the set of results to show with the ``-S`` or
310 ``--select`` flags along with the letters for each type (i.e., ``-S efps``
311 would select all results). By default ``analyze.py`` will use ``-S ef`` (i.e.,
312 [e]rrors and [f]ailures) unless the ``--search`` filter is given in which case
313 the default is to search all results (i.e., ``-S efps``).
315 One can find all results which contain a ``REGEXP``. To filter results using a
316 regular expression use the ``--search REGEXP`` option. In this case, by default,
317 all result types will be searched for a match against the given ``REGEXP``. If a
318 test result output contains a match it is selected into the set of results to show.
320 An example of using ``--search`` would be to search all tests results for some
321 log message, perhaps a warning or error.
323 Using XML Results File from CI
324 """"""""""""""""""""""""""""""
326 ``analyze.py`` actually only needs the ``topotests.xml`` file to run. This is
327 very useful for analyzing a CI run failure where one only need download the
328 ``topotests.xml`` artifact from the run and then pass that to ``analyze.py``
329 with the ``-r`` or ``--results`` option.
331 For local runs if you wish to simply copy the ``topotests.xml`` file (leaving
332 the log files where they are), you can pass the ``-a`` (or ``--save-xml``)
333 instead of the ``-A`` (or ``-save``) options.
335 Analyze Results from a Container Run
336 """"""""""""""""""""""""""""""""""""
338 ``analyze.py`` can also be used with ``docker`` or ``podman`` containers.
339 Everything works exactly as with a host run except that you specify the name of
340 the container, or the container-id, using the `-C` or ``--container`` option.
341 ``analyze.py`` will then use the results inside that containers
342 ``/tmp/topotests`` directory. It will extract and save those results when you
343 pass the ``-A`` or ``-a`` options just as withe host results.
352 sudo -E pytest ./test_to_be_run.py
354 For example, and assuming you are inside the frr directory:
358 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
359 sudo -E pytest ./test_bgp_l3vpn_to_bgp_vrf.py
361 For further options, refer to pytest documentation.
363 Test will set exit code which can be used with ``git bisect``.
365 For the simulated topology, see the description in the python file.
367 Running Topotests with AddressSanitizer
368 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
370 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
371 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
372 information on AddressSanitizer, see
373 https://github.com/google/sanitizers/wiki/AddressSanitizer.
375 The checks are done automatically in the library call of ``checkRouterRunning``
376 (ie at beginning of tests when there is a check for all daemons running). No
377 changes or extra configuration for topotests is required beside compiling the
378 suite with AddressSanitizer enabled.
380 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
381 If found, then this is added with context (calling test) to
382 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
384 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
385 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
386 for ``master`` branch:
390 git clone https://github.com/FRRouting/frr.git
394 --enable-address-sanitizer \
395 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
396 --localstatedir=/var/run/frr \
397 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
398 --with-moduledir=/usr/lib/frr/modules \
399 --enable-multipath=0 --enable-rtadv \
400 --enable-tcp-zebra --enable-fpm --enable-pimd \
404 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
405 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
407 and create ``frr`` user and ``frrvty`` group as shown above.
409 Debugging Topotest Failures
410 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
412 Install and run tests inside ``tmux`` or ``byobu`` for best results.
414 ``XTerm`` is also fully supported. GNU ``screen`` can be used in most
415 situations; however, it does not work as well with launching ``vtysh`` or shell
418 For the below debugging options which launch programs or CLIs, topotest should
419 be run within ``tmux`` (or ``screen``)_, as ``gdb``, the shell or ``vtysh`` will
420 be launched using that windowing program, otherwise ``xterm`` will be attempted
421 to launch the given programs.
423 NOTE: you must run the topotest (pytest) such that your DISPLAY, STY or TMUX
424 environment variables are carried over. You can do this by passing the
425 ``-E`` flag to ``sudo`` or you can modify your ``/etc/sudoers`` config to
426 automatically pass that environment variable through to the ``sudo``
429 .. _screen: https://www.gnu.org/software/screen/
430 .. _tmux: https://github.com/tmux/tmux/wiki
435 One can view and capture packets on any of the networks or interfaces defined by
436 the topotest by specifying the ``--pcap=NET|INTF|all[,NET|INTF,...]`` CLI option
437 as shown in the examples below.
441 # Capture on all networks in isis_topo1 test
442 sudo -E pytest isis_topo1 --pcap=all
444 # Capture on `sw1` network
445 sudo -E pytest isis_topo1 --pcap=sw1
447 # Capture on `sw1` network and on interface `eth0` on router `r2`
448 sudo -E pytest isis_topo1 --pcap=sw1,r2:r2-eth0
450 For each capture a window is opened displaying a live summary of the captured
451 packets. Additionally, the entire packet stream is captured in a pcap file in
452 the tests log directory e.g.,::
456 $ sudo -E pytest isis_topo1 --pcap=sw1,r2:r2-eth0
458 $ ls -l /tmp/topotests/isis_topo1.test_isis_topo1/
459 -rw------- 1 root root 45172 Apr 19 05:30 capture-r2-r2-eth0.pcap
460 -rw------- 1 root root 48412 Apr 19 05:30 capture-sw1.pcap
463 Viewing Live Daemon Logs
464 """"""""""""""""""""""""
466 One can live view daemon or the frr logs in separate windows using the
467 ``--logd`` CLI option as shown below.
471 # View `ripd` logs on all routers in test
472 sudo -E pytest rip_allow_ecmp --logd=ripd
474 # View `ripd` logs on all routers and `mgmtd` log on `r1`
475 sudo -E pytest rip_allow_ecmp --logd=ripd --logd=mgmtd,r1
477 For each capture a window is opened displaying a live summary of the captured
478 packets. Additionally, the entire packet stream is captured in a pcap file in
479 the tests log directory e.g.,::
481 When using a unified log file `frr.log` one substitutes `frr` for the daemon
482 name in the ``--logd`` CLI option, e.g.,
486 # View `frr` log on all routers in test
487 sudo -E pytest some_test_suite --logd=frr
489 Spawning Debugging CLI, ``vtysh`` or Shells on Routers on Test Failure
490 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
492 One can have a debugging CLI invoked on test failures by specifying the
493 ``--cli-on-error`` CLI option as shown in the example below.
497 sudo -E pytest --cli-on-error all-protocol-startup
499 The debugging CLI can run shell or vtysh commands on any combination of routers
500 It can also open shells or vtysh in their own windows for any combination of
501 routers. This is usually the most useful option when debugging failures. Here is
502 the help command from within a CLI launched on error:
506 test_bgp_multiview_topo1/test_bgp_routingTable> help
509 cli :: open a secondary CLI window
514 HOST can be a host or one of the following:
516 - '.' for the parent munet
517 - a regex specified between '/' (e.g., '/rtr.*/')
520 logd HOST [HOST ...] DAEMON :: tail -f on the logfile of the given DAEMON for the given HOST[S]
521 pcap NETWORK :: capture packets from NETWORK into file capture-NETWORK.pcap the command is run within a new window which also shows packet summaries. NETWORK can also be an interface specified as HOST:INTF. To capture inside the host namespace.
522 stderr HOST [HOST ...] DAEMON :: tail -f on the stderr of the given DAEMON for the given HOST[S]
523 stdlog HOST [HOST ...] :: tail -f on the `frr.log` for the given HOST[S]
524 stdout HOST [HOST ...] DAEMON :: tail -f on the stdout of the given DAEMON for the given HOST[S]
525 term HOST [HOST ...] :: open terminal[s] (TMUX or XTerm) on HOST[S], * for all
526 vtysh ROUTER [ROUTER ...] ::
527 xterm HOST [HOST ...] :: open XTerm[s] on HOST[S], * for all
529 [ROUTER ...] COMMAND :: execute vtysh COMMAND on the router[s]
530 [HOST ...] sh <SHELL-COMMAND> :: execute <SHELL-COMMAND> on hosts
531 [HOST ...] shi <INTERACTIVE-COMMAND> :: execute <INTERACTIVE-COMMAND> on HOST[s]
533 test_bgp_multiview_topo1/test_bgp_routingTable> r1 show int br
534 ------ Host: r1 ------
535 Interface Status VRF Addresses
536 --------- ------ --- ---------
541 r1-eth0 up default 172.16.1.254/24
542 r1-stub up default 172.20.0.1/28
544 ----------------------
545 test_bgp_multiview_topo1/test_bgp_routingTable>
547 Additionally, one can have ``vtysh`` or a shell launched on all routers when a
548 test fails. To launch the given process on each router after a test failure
549 specify one of ``--shell-on-error`` or ``--vtysh-on-error``.
551 Spawning ``vtysh`` or Shells on Routers
552 """""""""""""""""""""""""""""""""""""""
554 Topotest can automatically launch a shell or ``vtysh`` for any or all routers in
555 a test. This is enabled by specifying 1 of 2 CLI arguments ``--shell`` or
556 ``--vtysh``. Both of these options can be set to a single router value, multiple
557 comma-seperated values, or ``all``.
559 When either of these options are specified topotest will pause after setup and
560 each test to allow for inspection of the router state.
562 Here's an example of launching ``vtysh`` on routers ``rt1`` and ``rt2``.
566 sudo -E pytest --vtysh=rt1,rt2 all-protocol-startup
571 Topotest can automatically launch any daemon with ``gdb``, possibly setting
572 breakpoints for any test run. This is enabled by specifying 1 or 2 CLI arguments
573 ``--gdb-routers`` and ``--gdb-daemons``. Additionally ``--gdb-breakpoints`` can
574 be used to automatically set breakpoints in the launched ``gdb`` processes.
576 Each of these options can be set to a single value, multiple comma-seperated
577 values, or ``all``. If ``--gdb-routers`` is empty but ``--gdb_daemons`` is set
578 then the given daemons will be launched in ``gdb`` on all routers in the test.
579 Likewise if ``--gdb_routers`` is set, but ``--gdb_daemons`` is empty then all
580 daemons on the given routers will be launched in ``gdb``.
582 Here's an example of launching ``zebra`` and ``bgpd`` inside ``gdb`` on router
583 ``r1`` with a breakpoint set on ``nb_config_diff``
587 sudo -E pytest --gdb-routers=r1 \
588 --gdb-daemons=bgpd,zebra \
589 --gdb-breakpoints=nb_config_diff \
592 Reporting Memleaks with FRR Memory Statistics
593 """""""""""""""""""""""""""""""""""""""""""""
595 FRR reports all allocated FRR memory objects on exit to standard error.
596 Topotest can be run to report such output as errors in order to check for
597 memleaks in FRR memory allocations. Specifying the CLI argument
598 ``--memleaks`` will enable reporting FRR-based memory allocations at exit as errors.
602 sudo -E pytest --memleaks all-protocol-startup
605 StdErr log from daemos after exit
606 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
608 When running with ``--memleaks``, to enable the reporting of other,
609 non-memory related, messages seen on StdErr after the daemons exit,
610 the following env variable can be set::
612 export TOPOTESTS_CHECK_STDERR=Yes
614 (The value doesn't matter at this time. The check is whether the env
615 variable exists or not.) There is no pass/fail on this reporting; the
616 Output will be reported to the console.
618 Collect Memory Leak Information
619 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
621 When running with ``--memleaks``, FRR processes report unfreed memory
622 allocations upon exit. To enable also reporting of memory leaks to a specific
623 location, define an environment variable ``TOPOTESTS_CHECK_MEMLEAK`` with the
626 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
628 For tests that support the TOPOTESTS_CHECK_MEMLEAK environment variable, this
629 will enable output to the information to files with the given prefix (followed
631 file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
634 Detecting Memleaks with Valgrind
635 """"""""""""""""""""""""""""""""
637 Topotest can automatically launch all daemons with ``valgrind`` to check for
638 memleaks. This is enabled by specifying 1 or 2 CLI arguments.
639 ``--valgrind-memleaks`` will enable general memleak detection, and
640 ``--valgrind-extra`` enables extra functionality including generating a
641 suppression file. The suppression file ``tools/valgrind.supp`` is used when
642 memleak detection is enabled.
646 sudo -E pytest --valgrind-memleaks all-protocol-startup
648 Collecting Performance Data using perf(1)
649 """""""""""""""""""""""""""""""""""""""""
651 Topotest can automatically launch any daemon under ``perf(1)`` to collect
652 performance data. The daemon is run in non-daemon mode with ``perf record -g``.
653 The ``perf.data`` file will be saved in the router specific directory under the
656 Here's an example of collecting performance data from ``mgmtd`` on router ``r1``
657 during the config_timing test.
661 $ sudo -E pytest --perf=mgmtd,r1 config_timing
663 $ find /tmp/topotests/ -name '*perf.data*'
664 /tmp/topotests/config_timing.test_config_timing/r1/perf.data
666 To specify different arguments for ``perf record``, one can use the
667 ``--perf-options`` this will replace the ``-g`` used by default.
669 .. _topotests_docker:
671 Running Tests with Docker
672 -------------------------
674 There is a Docker image which allows to run topotests.
679 If you have Docker installed, you can run the topotests in Docker. The easiest
680 way to do this, is to use the make targets from this repository.
682 Your current user needs to have access to the Docker daemon. Alternatively you
683 can run these commands as root.
689 This command will pull the most recent topotests image from Dockerhub, compile
690 FRR inside of it, and run the topotests.
695 Internally, the topotests make target uses a shell script to pull the image and
696 spawn the Docker container.
698 There are several environment variables which can be used to modify the
699 behavior of the script, these can be listed by calling it with ``-h``:
703 ./tests/topotests/docker/frr-topotests.sh -h
705 For example, a volume is used to cache build artifacts between multiple runs of
706 the image. If you need to force a complete recompile, you can set
711 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
713 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
714 arguments and the first one starts with ``/`` or ``./``, they will replace the
715 call to pytest. If the appended arguments do not match this patttern, they will
716 be provided to pytest as arguments. So, to run a specific test with more
721 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
723 And to compile FRR but drop into a shell instead of running pytest:
727 ./tests/topotests/docker/frr-topotests.sh /bin/bash
732 The Docker image just includes all the components to run the topotests, but not
733 the topotests themselves. So if you just want to write tests and don't want to
734 make changes to the environment provided by the Docker image. You don't need to
735 build your own Docker image if you do not want to.
737 When developing new tests, there is one caveat though: The startup script of
738 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
739 pollution of the container with build artefacts from the host. This will also
740 result in your newly written tests being unavailable in the container unless at
741 least added to the index with ``git-add``.
743 If you do want to test changes to the Docker image, you can locally build the
744 image and run the tests without pulling from the registry using the following
750 TOPOTEST_PULL=0 make topotests
753 .. _topotests-guidelines:
761 To run the whole suite of tests the following commands must be executed at the
762 top level directory of topotest:
766 $ # Change to the top level directory of topotests.
767 $ cd path/to/topotests
768 $ # Tests must be run as root, since micronet requires it.
771 In order to run a specific test, you can use the following command:
775 $ # running a specific topology
776 $ sudo -E pytest ospf-topo1/
777 $ # or inside the test folder
779 $ sudo -E pytest # to run all tests inside the directory
780 $ sudo -E pytest test_ospf_topo1.py # to run a specific test
781 $ # or outside the test folder
783 $ sudo -E pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
785 The output of the tested daemons will be available at the temporary folder of
790 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
792 zebra.err # zebra stderr output
793 zebra.log # zebra log file
794 zebra.out # zebra stdout output
797 You can also run memory leak tests to get reports:
801 $ # Set the environment variable to apply to a specific test...
802 $ sudo -E env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
803 $ # ...or apply to all tests adding this line to the configuration file
804 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
805 $ # You can also use your editor
807 $ # After running tests you should see your files:
808 $ ls /tmp/memleak_report_*
809 memleak_report_test_ospf_topo1.txt
814 This section will guide you in all recommended steps to produce a standard
817 This is the recommended test writing routine:
819 - Write a topology (Graphviz recommended)
820 - Obtain configuration files
821 - Write the test itself
822 - Format the new code using `black <https://github.com/psf/black>`_
823 - Create a Pull Request
825 Some things to keep in mind:
827 - BGP tests MUST use generous convergence timeouts - you must ensure
828 that any test involving BGP uses a convergence timeout of at least
830 - Topotests are run on a range of Linux versions: if your test
831 requires some OS-specific capability (like mpls support, or vrf
832 support), there are test functions available in the libraries that
833 will help you determine whether your test should run or be skipped.
834 - Avoid including unstable data in your test: don't rely on link-local
835 addresses or ifindex values, for example, because these can change
837 - Using sleep is almost never appropriate. As an example: if the test resets the
838 peers in BGP, the test should look for the peers re-converging instead of just
839 sleeping an arbitrary amount of time and continuing on. See
840 ``verify_bgp_convergence`` as a good example of this. In particular look at
841 it's use of the ``@retry`` decorator. If you are having troubles figuring out
842 what to look for, please do not be afraid to ask.
843 - Don't duplicate effort. There exists many protocol utility functions that can
844 be found in their eponymous module under ``tests/topotests/lib/`` (e.g.,
849 Topotest File Hierarchy
850 """""""""""""""""""""""
852 Before starting to write any tests one must know the file hierarchy. The
853 repository hierarchy looks like this:
857 $ cd path/to/topotest
860 ./README.md # repository read me
861 ./GUIDELINES.md # this file
862 ./conftest.py # test hooks - pytest related functions
863 ./example-test # example test folder
864 ./example-test/__init__.py # python package marker - must always exist.
865 ./example-test/test_template.jpg # generated topology picture - see next section
866 ./example-test/test_template.dot # Graphviz dot file
867 ./example-test/test_template.py # the topology plus the test
869 ./ospf-topo1 # the ospf topology test
870 ./ospf-topo1/r1 # router 1 configuration files
871 ./ospf-topo1/r1/zebra.conf # zebra configuration file
872 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
873 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
874 # removed other for shortness sake
876 ./lib # shared test/topology functions
877 ./lib/topogen.py # topogen implementation
878 ./lib/topotest.py # topotest implementation
880 Guidelines for creating/editing topotest:
882 - New topologies that don't fit the existing directories should create its own
883 - Always remember to add the ``__init__.py`` to new folders, this makes auto
884 complete engines and pylint happy
885 - Router (Quagga/FRR) specific code should go on topotest.py
886 - Generic/repeated router actions should have an abstraction in
888 - Generic/repeated non-router code should go to topotest.py
889 - pytest related code should go to conftest.py (e.g. specialized asserts)
891 Defining the Topology
892 """""""""""""""""""""
894 The first step to write a new test is to define the topology. This step can be
895 done in many ways, but the recommended is to use Graphviz to generate a drawing
896 of the topology. It allows us to see the topology graphically and to see the
897 names of equipment, links and addresses.
899 Here is an example of Graphviz dot file that generates the template topology
900 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
901 get outdated, please see the linked file)::
923 label="s1\n192.168.0.0/24",
929 label="s2\n192.168.1.0/24",
935 r1 -- s1 [label="eth0\n.1"];
937 r1 -- s2 [label="eth1\n.100"];
938 r2 -- s2 [label="eth0\n.1"];
941 Here is the produced graph:
965 label="s1\n192.168.0.0/24",
971 label="s2\n192.168.1.0/24",
977 r1 -- s1 [label="eth0\n.1"];
979 r1 -- s2 [label="eth1\n.100"];
980 r2 -- s2 [label="eth0\n.1"];
983 Generating / Obtaining Configuration Files
984 """"""""""""""""""""""""""""""""""""""""""
986 In order to get the configuration files or command output for each router, we
987 need to run the topology and execute commands in ``vtysh``. The quickest way to
988 achieve that is writing the topology building code and running the topology.
990 To bootstrap your test topology, do the following steps:
992 - Copy the template test
997 $ touch new-topo/__init__.py
998 $ cp example-test/test_template.py new-topo/test_new_topo.py
1000 - Modify the template according to your dot file
1002 Here is the template topology described in the previous section in python code:
1011 If more specialized topology definitions, or router initialization arguments are
1012 required a build function can be used instead of a dictionary:
1016 def build_topo(tgen):
1020 for routern in range(1, 3):
1021 tgen.add_router("r{}".format(routern))
1023 # Create a switch with just one router connected to it to simulate a
1025 switch = tgen.add_switch("s1")
1026 switch.add_link(tgen.gears["r1"])
1028 # Create a connection between r1 and r2
1029 switch = tgen.add_switch("s2")
1030 switch.add_link(tgen.gears["r1"])
1031 switch.add_link(tgen.gears["r2"])
1035 Topogen allows us to run the topology without running any tests, you can do
1036 that using the following example commands:
1040 $ # Running your bootstraped topology
1041 $ sudo -E pytest -s --topology-only new-topo/test_new_topo.py
1042 $ # Running the test_template.py topology
1043 $ sudo -E pytest -s --topology-only example-test/test_template.py
1044 $ # Running the ospf_topo1.py topology
1045 $ sudo -E pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
1047 Parameters explanation:
1053 Actives input/output capture. If this is not specified a new window will be
1054 opened for the interactive CLI, otherwise it will be activated inline.
1056 .. option:: --topology-only
1058 Don't run any tests, just build the topology.
1060 After executing the commands above, you should get the following terminal
1065 frr/tests/topotests# sudo -E pytest -s --topology-only ospf_topo1/test_ospf_topo1.py
1066 ============================= test session starts ==============================
1067 platform linux -- Python 3.9.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
1068 rootdir: /home/chopps/w/frr/tests/topotests, configfile: pytest.ini
1069 plugins: forked-1.3.0, xdist-2.3.0
1075 The last line shows us that we are now using the CLI (Command Line
1076 Interface), from here you can call your router ``vtysh`` or even bash.
1078 Here's the help text:
1086 sh [hosts] <shell-command> :: execute <shell-command> on <host>
1087 term [hosts] :: open shell terminals for hosts
1088 vtysh [hosts] :: open vtysh terminals for hosts
1089 [hosts] <vtysh-command> :: execute vtysh-command on hosts
1091 Here are some commands example:
1095 unet> sh r1 ping 10.0.3.1
1096 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
1097 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
1098 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
1099 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
1101 --- 10.0.3.1 ping statistics ---
1102 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
1103 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
1106 Building configuration...
1108 Current configuration:
1110 frr version 8.1-dev-my-manual-build
1111 frr defaults traditional
1113 log file /tmp/topotests/ospf_topo1.test_ospf_topo1/r1/zebra.log
1118 ------ Host: r1 ------
1119 zebra ospfd ospf6d staticd
1120 ------- End: r1 ------
1121 ------ Host: r2 ------
1122 zebra ospfd ospf6d staticd
1123 ------- End: r2 ------
1124 ------ Host: r3 ------
1125 zebra ospfd ospf6d staticd
1126 ------- End: r3 ------
1127 ------ Host: r4 ------
1128 zebra ospfd ospf6d staticd
1129 ------- End: r4 ------
1131 After you successfully configured your topology, you can obtain the
1132 configuration files (per-daemon) using the following commands:
1136 unet> sh r3 vtysh -d ospfd
1138 Hello, this is FRRouting (version 3.1-devrzalamena-build).
1139 Copyright 1996-2005 Kunihiro Ishiguro, et al.
1141 r1# show running-config
1142 Building configuration...
1144 Current configuration:
1146 frr version 3.1-devrzalamena-build
1147 frr defaults traditional
1148 no service integrated-vtysh-config
1153 ospf router-id 10.0.255.3
1155 redistribute connected
1157 network 10.0.3.0/24 area 0
1158 network 10.0.10.0/24 area 0
1159 network 172.16.0.0/24 area 1
1166 You can also login to the node specified by nsenter using bash, etc.
1167 A pid file for each node will be created in the relevant test dir.
1168 You can run scripts inside the node, or use vtysh's <tab> or <?> feature.
1173 # cd tests/topotests/srv6_locator
1174 # ./test_srv6_locator.py --topology-only
1175 unet> r1 show segment-routing srv6 locator
1177 Name ID Prefix Status
1178 -------------------- ------- ------------------------ -------
1179 loc1 1 2001:db8:1:1::/64 Up
1180 loc2 2 2001:db8:2:2::/64 Up
1183 # nsenter -a -t $(cat /tmp/topotests/srv6_locator.test_srv6_locator/r1.pid) bash --norc
1185 r1# r1 show segment-routing srv6 locator
1187 Name ID Prefix Status
1188 -------------------- ------- ------------------------ -------
1189 loc1 1 2001:db8:1:1::/64 Up
1190 loc2 2 2001:db8:2:2::/64 Up
1195 Test topologies should always be bootstrapped from
1196 :file:`tests/topotests/example_test/test_template.py` because it contains
1197 important boilerplate code that can't be avoided, like:
1203 # For all routers arrange for:
1204 # - starting zebra using config file from <rtrname>/zebra.conf
1205 # - starting ospfd using an empty config file.
1206 for rname, router in router_list.items():
1207 router.load_config(TopoRouter.RD_ZEBRA, "zebra.conf")
1208 router.load_config(TopoRouter.RD_OSPF)
1211 - The topology definition or build function
1220 def build_topo(tgen):
1221 # topology build code
1224 - pytest setup/teardown fixture to start the topology and supply ``tgen``
1230 @pytest.fixture(scope="module")
1232 "Setup/Teardown the environment and provide tgen argument to tests"
1234 tgen = Topogen(topodef, module.__name__)
1236 tgen = Topogen(build_topo, module.__name__)
1240 # Start and configure the router daemons
1243 # Provide tgen as argument to each test function
1246 # Teardown after last test runs
1247 tgen.stop_topology()
1252 - Directory name for a new topotest must not contain hyphen (``-``) characters.
1253 To separate words, use underscores (``_``). For example, ``tests/topotests/bgp_new_example``.
1254 - Test code should always be declared inside functions that begin with the
1255 ``test_`` prefix. Functions beginning with different prefixes will not be run
1257 - Configuration files and long output commands should go into separated files
1258 inside folders named after the equipment.
1259 - Tests must be able to run without any interaction. To make sure your test
1260 conforms with this, run it without the :option:`-s` parameter.
1261 - Use `black <https://github.com/psf/black>`_ code formatter before creating
1262 a pull request. This ensures we have a unified code style.
1263 - Mark test modules with pytest markers depending on the daemons used during the
1264 tests (see :ref:`topotests-markers`)
1265 - Always use IPv4 :rfc:`5737` (``192.0.2.0/24``, ``198.51.100.0/24``,
1266 ``203.0.113.0/24``) and IPv6 :rfc:`3849` (``2001:db8::/32``) ranges reserved
1271 - Keep results in stack variables, so people inspecting code with ``pdb`` can
1272 easily print their values.
1278 assert foobar(router1, router2)
1284 result = foobar(router1, router2)
1287 - Use ``assert`` messages to indicate where the test failed.
1293 for router in router_list:
1295 assert condition, 'Router "{}" condition failed'.format(router.name)
1300 The most effective ways to inspect topology tests are:
1302 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
1303 appear when an assertion fails
1305 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
1307 - Set a breakpoint in the test code with ``pdb``
1313 # Add the pdb import at the beginning of the file
1317 # Add a breakpoint where you think the problem is
1323 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
1324 shell allows us to run many useful operations like:
1326 - Setting breaking point on file/function/conditions (e.g. ``break``,
1328 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
1329 - Running python code
1333 The TopoGear (equipment abstraction class) implements the ``__str__`` method
1334 that allows the user to inspect equipment information.
1336 Example of pdb usage:
1340 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
1341 -> for rnum in range(1, 5):
1343 Documented commands (type help <topic>):
1344 ========================================
1345 EOF bt cont enable jump pp run unt
1346 a c continue exit l q s until
1347 alias cl d h list quit step up
1348 args clear debug help n r tbreak w
1349 b commands disable ignore next restart u whatis
1350 break condition down j p return unalias where
1352 Miscellaneous help topics:
1353 ==========================
1356 Undocumented commands:
1357 ======================
1361 116 title2="Expected output")
1363 118 def test_ospf_convergence():
1364 119 "Test OSPF daemon convergence"
1366 121 -> for rnum in range(1, 5):
1367 122 router = 'r{}'.format(rnum)
1369 124 # Load expected results from the command
1370 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1371 126 expected = open(reffile).read()
1373 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
1374 -> router = 'r{}'.format(rnum)
1376 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
1377 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1382 (Pdb) tgen = get_topogen()
1383 (Pdb) pp tgen.gears[router]
1384 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
1385 (Pdb) pp str(tgen.gears[router])
1386 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
1389 121 for rnum in range(1, 5):
1390 122 router = 'r{}'.format(rnum)
1392 124 # Load expected results from the command
1393 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1394 126 expected = open(reffile).read()
1396 128 # Run test function until we get an result. Wait at most 60 seconds.
1397 129 test_func = partial(compare_show_ip_ospf, router, expected)
1398 130 result, diff = topotest.run_and_expect(test_func, '',
1399 (Pdb) router1 = tgen.gears[router]
1400 (Pdb) router1.vtysh_cmd('show ip ospf route')
1401 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
1405 To enable more debug messages in other Topogen subsystems, more
1406 logging messages can be displayed by modifying the test configuration file
1412 # Change the default verbosity line from 'info'...
1417 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
1418 To learn/remember common code snippets see :ref:`topotests-snippets`.
1420 Before creating a new topology, make sure that there isn't one already that
1421 does what you need. If nothing is similar, then you may create a new topology,
1422 preferably, using the newest template
1423 (:file:`tests/topotests/example-test/test_template.py`).
1425 .. include:: topotests-markers.rst
1427 .. include:: topotests-snippets.rst
1432 All the configs and scripts are licensed under a ISC-style license. See Python
1433 scripts for details.