6 Topotests is a suite of topology tests for FRR built on top of micronet.
11 Topotests run under python3. Additionally, for ExaBGP (which is used
12 in some of the BGP tests) an older python2 version (and the python2
13 version of ``pip``) must be installed.
15 Tested with Ubuntu 20.04,Ubuntu 18.04, and Debian 11.
17 Instructions are the same for all setups (i.e. ExaBGP is only used for
20 Installing Topotest Requirements
21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
26 apt-get install iproute2
27 apt-get install net-tools
28 apt-get install python3-pip
29 python3 -m pip install wheel
30 python3 -m pip install 'pytest>=6.2.4'
31 python3 -m pip install 'pytest-xdist>=2.3.0'
32 python3 -m pip install 'scapy>=2.4.5'
33 python3 -m pip install xmltodict
34 # Use python2 pip to install older ExaBGP
35 python2 -m pip install 'exabgp<4.0.0'
36 useradd -d /var/run/exabgp/ -s /bin/false exabgp
38 # To enable the gRPC topotest install:
39 python3 -m pip install grpcio grpcio-tools
41 # Install Socat tool to run PIMv6 tests,
42 # Socat code can be taken from below url,
43 # which has latest changes done for PIMv6,
45 https://github.com/opensourcerouting/socat/
51 Optional, will give better output.
55 disable apport (which move core files)
57 Set ``enabled=0`` in ``/etc/default/apport``.
59 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
61 #<domain> <type> <item> <value>
63 root soft core unlimited
65 root hard core unlimited
67 Reboot for options to take effect.
69 SNMP Utilities Installation
70 """""""""""""""""""""""""""
72 To run SNMP test you need to install SNMP utilities and MIBs. Unfortunately
73 there are some errors in the upstream MIBS which need to be patched up. The
74 following steps will get you there on Ubuntu 20.04.
78 apt install libsnmp-dev
79 apt install snmpd snmp
80 apt install snmp-mibs-downloader
82 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/iana/IANA-IPPM-METRICS-REGISTRY-MIB -O /usr/share/snmp/mibs/iana/IANA-IPPM-METRICS-REGISTRY-MIB
83 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/ietf/SNMPv2-PDU -O /usr/share/snmp/mibs/ietf/SNMPv2-PDU
84 wget https://raw.githubusercontent.com/FRRouting/frr-mibs/main/ietf/IPATM-IPMC-MIB -O /usr/share/snmp/mibs/ietf/IPATM-IPMC-MIB
85 edit /etc/snmp/snmp.conf to look like this
86 # As the snmp packages come without MIB files due to license reasons, loading
87 # of MIBs is disabled by default. If you added the MIBs you can reenable
88 # loading them by commenting out the following line.
95 FRR needs to be installed separately. It is assume to be configured like the
96 standard Ubuntu Packages:
98 - Binaries in :file:`/usr/lib/frr`
99 - State Directory :file:`/var/run/frr`
100 - Running under user ``frr``, group ``frr``
101 - vtygroup: ``frrvty``
102 - config directory: :file:`/etc/frr`
103 - For FRR Packages, install the dbg package as well for coredump decoding
105 No FRR config needs to be done and no FRR daemons should be run ahead of the
106 test. They are all started as part of the test.
111 If you prefer to manually build FRR, then use the following suggested config:
117 --localstatedir=/var/run/frr \
118 --sbindir=/usr/lib/frr \
119 --sysconfdir=/etc/frr \
123 --enable-multipath=64 \
126 --enable-vty-group=frrvty \
127 --enable-snmp=agentx \
128 --with-pkg-extra-version=-my-manual-build
130 And create ``frr`` user and ``frrvty`` group as follows:
134 addgroup --system --gid 92 frr
135 addgroup --system --gid 85 frrvty
136 adduser --system --ingroup frr --home /var/run/frr/ \
137 --gecos "FRRouting suite" --shell /bin/false frr
138 usermod -G frrvty frr
143 Configure your sudo environment
144 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
146 Topotests must be run as root. Normally this will be accomplished through the
147 use of the ``sudo`` command. In order for topotests to be able to open new
148 windows (either XTerm or byobu/screen/tmux windows) certain environment
149 variables must be passed through the sudo command. One way to do this is to
150 specify the ``-E`` flag to ``sudo``. This will carry over most if not all
151 your environment variables include ``PATH``. For example:
155 sudo -E python3 -m pytest -s -v
157 If you do not wish to use ``-E`` (e.g., to avoid ``sudo`` inheriting
158 ``PATH``) you can modify your `/etc/sudoers` config file to specifically pass
159 the environment variables required by topotests. Add the following commands to
160 your ``/etc/sudoers`` config file.
164 Defaults env_keep="TMUX"
165 Defaults env_keep+="TMUX_PANE"
166 Defaults env_keep+="STY"
167 Defaults env_keep+="DISPLAY"
169 If there was already an ``env_keep`` configuration there be sure to use the
170 ``+=`` rather than ``=`` on the first line above as well.
173 Execute all tests in distributed test mode
174 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
178 sudo -E pytest -s -v -nauto --dist=loadfile
180 The above command must be executed from inside the topotests directory.
182 All test\_\* scripts in subdirectories are detected and executed (unless
183 disabled in ``pytest.ini`` file). Pytest will execute up to N tests in parallel
184 where N is based on the number of cores on the host.
186 Analyze Test Results (``analyze.py``)
187 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
189 By default router and execution logs are saved in ``/tmp/topotests`` and an XML
190 results file is saved in ``/tmp/topotests.xml``. An analysis tool ``analyze.py``
191 is provided to archive and analyze these results after the run completes.
193 After the test run completes one should pick an archive directory to store the
194 results in and pass this value to ``analyze.py``. On first execution the results
195 are copied to that directory from ``/tmp``, and subsequent runs use that
196 directory for analyzing the results. Below is an example of this which also
197 shows the default behavior which is to display all failed and errored tests in
202 ~/frr/tests/topotests# ./analyze.py -Ar run-save
203 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge
204 ospf_basic_functionality/test_ospf_lan.py::test_ospf_lan_tc1_p0
205 bgp_gr_functionality_topo2/test_bgp_gr_functionality_topo2.py::test_BGP_GR_10_p2
206 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_routingTable
208 Here we see that 4 tests have failed. We an dig deeper by displaying the
209 captured logs and errors. First let's redisplay the results enumerated by adding
214 ~/frr/tests/topotests# ./analyze.py -Ar run-save -E
215 0 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge
216 1 ospf_basic_functionality/test_ospf_lan.py::test_ospf_lan_tc1_p0
217 2 bgp_gr_functionality_topo2/test_bgp_gr_functionality_topo2.py::test_BGP_GR_10_p2
218 3 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_routingTable
220 Now to look at the error message for a failed test we use ``-T N`` where N is
221 the number of the test we are interested in along with ``--errmsg`` option.
225 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0 --errmsg
226 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge: AssertionError: BGP did not converge:
228 IPv4 Unicast Summary (VIEW 1):
229 BGP router identifier 172.30.1.1, local AS number 100 vrf-id -1
231 RIB entries 1, using 184 bytes of memory
232 Peers 3, using 2169 KiB of memory
234 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
235 172.16.1.1 4 65001 0 0 0 0 0 never Connect 0 N/A
236 172.16.1.2 4 65002 0 0 0 0 0 never Connect 0 N/A
237 172.16.1.5 4 65005 0 0 0 0 0 never Connect 0 N/A
239 Total number of neighbors 3
243 Now to look at the full text of the error for a failed test we use ``-T N``
244 where N is the number of the test we are interested in along with ``--errtext``
249 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0 --errtext
250 bgp_multiview_topo1/test_bgp_multiview_topo1.py::test_bgp_converge: def test_bgp_converge():
251 "Check for BGP converged on all peers and BGP views"
257 # Bail out with error if a router fails to converge
258 bgpStatus = net["r%s" % i].cmd('vtysh -c "show ip bgp view %s summary"' % view)
259 > assert False, "BGP did not converge:\n%s" % bgpStatus
260 E AssertionError: BGP did not converge:
262 E IPv4 Unicast Summary (VIEW 1):
263 E BGP router identifier 172.30.1.1, local AS number 100 vrf-id -1
265 E Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
266 E 172.16.1.1 4 65001 0 0 0 0 0 never Connect 0 N/A
267 E 172.16.1.2 4 65002 0 0 0 0 0 never Connect 0 N/A
270 To look at the full capture for a test including the stdout and stderr which
271 includes full debug logs, just use the ``-T N`` option without the ``--errmsg``
272 or ``--errtext`` options.
276 ~/frr/tests/topotests# ./analyze.py -Ar run-save -T0
277 @classname: bgp_multiview_topo1.test_bgp_multiview_topo1
278 @name: test_bgp_converge
280 @message: AssertionError: BGP did not converge:
282 system-out: --------------------------------- Captured Log ---------------------------------
283 2021-08-09 02:55:06,581 DEBUG: lib.micronet_compat.topo: Topo(unnamed): Creating
284 2021-08-09 02:55:06,581 DEBUG: lib.micronet_compat.topo: Topo(unnamed): addHost r1
286 2021-08-09 02:57:16,932 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary" 2> /dev/null | grep ^[0-9] | grep -vP " 11\\s+(\\d+)"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
287 2021-08-09 02:57:22,290 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary" 2> /dev/null | grep ^[0-9] | grep -vP " 11\\s+(\\d+)"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
288 2021-08-09 02:57:27,636 DEBUG: topolog.r1: LinuxNamespace(r1): cmd_status("['/bin/bash', '-c', 'vtysh -c "show ip bgp view 1 summary"']", kwargs: {'encoding': 'utf-8', 'stdout': -1, 'stderr': -2, 'shell': False})
289 --------------------------------- Captured Out ---------------------------------
290 system-err: --------------------------------- Captured Err ---------------------------------
301 For example, and assuming you are inside the frr directory:
305 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
306 ./test_bgp_l3vpn_to_bgp_vrf.py
308 For further options, refer to pytest documentation.
310 Test will set exit code which can be used with ``git bisect``.
312 For the simulated topology, see the description in the python file.
314 StdErr log from daemos after exit
315 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
317 To enable the reporting of any messages seen on StdErr after the daemons exit,
318 the following env variable can be set::
320 export TOPOTESTS_CHECK_STDERR=Yes
322 (The value doesn't matter at this time. The check is whether the env
323 variable exists or not.) There is no pass/fail on this reporting; the
324 Output will be reported to the console.
326 Collect Memory Leak Information
327 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
329 FRR processes can report unfreed memory allocations upon exit. To
330 enable the reporting of memory leaks, define an environment variable
331 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
333 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
335 This will enable the check and output to console and the writing of
336 the information to files with the given prefix (followed by testname),
337 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
340 Running Topotests with AddressSanitizer
341 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
343 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
344 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
345 information on AddressSanitizer, see
346 https://github.com/google/sanitizers/wiki/AddressSanitizer.
348 The checks are done automatically in the library call of ``checkRouterRunning``
349 (ie at beginning of tests when there is a check for all daemons running). No
350 changes or extra configuration for topotests is required beside compiling the
351 suite with AddressSanitizer enabled.
353 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
354 If found, then this is added with context (calling test) to
355 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
357 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
358 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
359 for ``master`` branch:
363 git clone https://github.com/FRRouting/frr.git
367 --enable-address-sanitizer \
368 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
369 --localstatedir=/var/run/frr \
370 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
371 --with-moduledir=/usr/lib/frr/modules \
372 --enable-multipath=0 --enable-rtadv \
373 --enable-tcp-zebra --enable-fpm --enable-pimd \
377 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
378 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
380 and create ``frr`` user and ``frrvty`` group as shown above.
382 Debugging Topotest Failures
383 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
385 Install and run tests inside ``tmux`` or ``byobu`` for best results.
387 ``XTerm`` is also fully supported. GNU ``screen`` can be used in most
388 situations; however, it does not work as well with launching ``vtysh`` or shell
391 For the below debugging options which launch programs or CLIs, topotest should
392 be run within ``tmux`` (or ``screen``)_, as ``gdb``, the shell or ``vtysh`` will
393 be launched using that windowing program, otherwise ``xterm`` will be attempted
394 to launch the given programs.
396 NOTE: you must run the topotest (pytest) such that your DISPLAY, STY or TMUX
397 environment variables are carried over. You can do this by passing the
398 ``-E`` flag to ``sudo`` or you can modify your ``/etc/sudoers`` config to
399 automatically pass that environment variable through to the ``sudo``
402 .. _screen: https://www.gnu.org/software/screen/
403 .. _tmux: https://github.com/tmux/tmux/wiki
405 Spawning Debugging CLI, ``vtysh`` or Shells on Routers on Test Failure
406 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
408 One can have a debugging CLI invoked on test failures by specifying the
409 ``--cli-on-error`` CLI option as shown in the example below.
413 sudo -E pytest --cli-on-error all-protocol-startup
415 The debugging CLI can run shell or vtysh commands on any combination of routers
416 It can also open shells or vtysh in their own windows for any combination of
417 routers. This is usually the most useful option when debugging failures. Here is
418 the help command from within a CLI launched on error:
422 test_bgp_multiview_topo1/test_bgp_routingTable> help
426 sh [hosts] <shell-command> :: execute <shell-command> on <host>
427 term [hosts] :: open shell terminals for hosts
428 vtysh [hosts] :: open vtysh terminals for hosts
429 [hosts] <vtysh-command> :: execute vtysh-command on hosts
431 test_bgp_multiview_topo1/test_bgp_routingTable> r1 show int br
432 ------ Host: r1 ------
433 Interface Status VRF Addresses
434 --------- ------ --- ---------
439 r1-eth0 up default 172.16.1.254/24
440 r1-stub up default 172.20.0.1/28
442 ----------------------
443 test_bgp_multiview_topo1/test_bgp_routingTable>
445 Additionally, one can have ``vtysh`` or a shell launched on all routers when a
446 test fails. To launch the given process on each router after a test failure
447 specify one of ``--shell-on-error`` or ``--vtysh-on-error``.
449 Spawning ``vtysh`` or Shells on Routers
450 """""""""""""""""""""""""""""""""""""""
452 Topotest can automatically launch a shell or ``vtysh`` for any or all routers in
453 a test. This is enabled by specifying 1 of 2 CLI arguments ``--shell`` or
454 ``--vtysh``. Both of these options can be set to a single router value, multiple
455 comma-seperated values, or ``all``.
457 When either of these options are specified topotest will pause after setup and
458 each test to allow for inspection of the router state.
460 Here's an example of launching ``vtysh`` on routers ``rt1`` and ``rt2``.
464 sudo -E pytest --vtysh=rt1,rt2 all-protocol-startup
469 Topotest can automatically launch any daemon with ``gdb``, possibly setting
470 breakpoints for any test run. This is enabled by specifying 1 or 2 CLI arguments
471 ``--gdb-routers`` and ``--gdb-daemons``. Additionally ``--gdb-breakpoints`` can
472 be used to automatically set breakpoints in the launched ``gdb`` processes.
474 Each of these options can be set to a single value, multiple comma-seperated
475 values, or ``all``. If ``--gdb-routers`` is empty but ``--gdb_daemons`` is set
476 then the given daemons will be launched in ``gdb`` on all routers in the test.
477 Likewise if ``--gdb_routers`` is set, but ``--gdb_daemons`` is empty then all
478 daemons on the given routers will be launched in ``gdb``.
480 Here's an example of launching ``zebra`` and ``bgpd`` inside ``gdb`` on router
481 ``r1`` with a breakpoint set on ``nb_config_diff``
485 sudo -E pytest --gdb-routers=r1 \
486 --gdb-daemons=bgpd,zebra \
487 --gdb-breakpoints=nb_config_diff \
490 Detecting Memleaks with Valgrind
491 """"""""""""""""""""""""""""""""
493 Topotest can automatically launch all daemons with ``valgrind`` to check for
494 memleaks. This is enabled by specifying 1 or 2 CLI arguments.
495 ``--valgrind-memleaks`` will enable general memleak detection, and
496 ``--valgrind-extra`` enables extra functionality including generating a
497 suppression file. The suppression file ``tools/valgrind.supp`` is used when
498 memleak detection is enabled.
502 sudo -E pytest --valgrind-memleaks all-protocol-startup
504 .. _topotests_docker:
506 Running Tests with Docker
507 -------------------------
509 There is a Docker image which allows to run topotests.
514 If you have Docker installed, you can run the topotests in Docker. The easiest
515 way to do this, is to use the make targets from this repository.
517 Your current user needs to have access to the Docker daemon. Alternatively you
518 can run these commands as root.
524 This command will pull the most recent topotests image from Dockerhub, compile
525 FRR inside of it, and run the topotests.
530 Internally, the topotests make target uses a shell script to pull the image and
531 spawn the Docker container.
533 There are several environment variables which can be used to modify the
534 behavior of the script, these can be listed by calling it with ``-h``:
538 ./tests/topotests/docker/frr-topotests.sh -h
540 For example, a volume is used to cache build artifacts between multiple runs of
541 the image. If you need to force a complete recompile, you can set
546 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
548 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
549 arguments and the first one starts with ``/`` or ``./``, they will replace the
550 call to pytest. If the appended arguments do not match this patttern, they will
551 be provided to pytest as arguments. So, to run a specific test with more
556 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
558 And to compile FRR but drop into a shell instead of running pytest:
562 ./tests/topotests/docker/frr-topotests.sh /bin/bash
567 The Docker image just includes all the components to run the topotests, but not
568 the topotests themselves. So if you just want to write tests and don't want to
569 make changes to the environment provided by the Docker image. You don't need to
570 build your own Docker image if you do not want to.
572 When developing new tests, there is one caveat though: The startup script of
573 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
574 pollution of the container with build artefacts from the host. This will also
575 result in your newly written tests being unavailable in the container unless at
576 least added to the index with ``git-add``.
578 If you do want to test changes to the Docker image, you can locally build the
579 image and run the tests without pulling from the registry using the following
585 TOPOTEST_PULL=0 make topotests
588 .. _topotests-guidelines:
596 To run the whole suite of tests the following commands must be executed at the
597 top level directory of topotest:
601 $ # Change to the top level directory of topotests.
602 $ cd path/to/topotests
603 $ # Tests must be run as root, since micronet requires it.
606 In order to run a specific test, you can use the following command:
610 $ # running a specific topology
611 $ sudo -E pytest ospf-topo1/
612 $ # or inside the test folder
614 $ sudo -E pytest # to run all tests inside the directory
615 $ sudo -E pytest test_ospf_topo1.py # to run a specific test
616 $ # or outside the test folder
618 $ sudo -E pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
620 The output of the tested daemons will be available at the temporary folder of
625 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
627 zebra.err # zebra stderr output
628 zebra.log # zebra log file
629 zebra.out # zebra stdout output
632 You can also run memory leak tests to get reports:
636 $ # Set the environment variable to apply to a specific test...
637 $ sudo -E env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
638 $ # ...or apply to all tests adding this line to the configuration file
639 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
640 $ # You can also use your editor
642 $ # After running tests you should see your files:
643 $ ls /tmp/memleak_report_*
644 memleak_report_test_ospf_topo1.txt
649 This section will guide you in all recommended steps to produce a standard
652 This is the recommended test writing routine:
654 - Write a topology (Graphviz recommended)
655 - Obtain configuration files
656 - Write the test itself
657 - Format the new code using `black <https://github.com/psf/black>`_
658 - Create a Pull Request
660 Some things to keep in mind:
662 - BGP tests MUST use generous convergence timeouts - you must ensure
663 that any test involving BGP uses a convergence timeout of at least
665 - Topotests are run on a range of Linux versions: if your test
666 requires some OS-specific capability (like mpls support, or vrf
667 support), there are test functions available in the libraries that
668 will help you determine whether your test should run or be skipped.
669 - Avoid including unstable data in your test: don't rely on link-local
670 addresses or ifindex values, for example, because these can change
672 - Using sleep is almost never appropriate. As an example: if the test resets the
673 peers in BGP, the test should look for the peers re-converging instead of just
674 sleeping an arbitrary amount of time and continuing on. See
675 ``verify_bgp_convergence`` as a good example of this. In particular look at
676 it's use of the ``@retry`` decorator. If you are having troubles figuring out
677 what to look for, please do not be afraid to ask.
678 - Don't duplicate effort. There exists many protocol utility functions that can
679 be found in their eponymous module under ``tests/topotests/lib/`` (e.g.,
684 Topotest File Hierarchy
685 """""""""""""""""""""""
687 Before starting to write any tests one must know the file hierarchy. The
688 repository hierarchy looks like this:
692 $ cd path/to/topotest
695 ./README.md # repository read me
696 ./GUIDELINES.md # this file
697 ./conftest.py # test hooks - pytest related functions
698 ./example-test # example test folder
699 ./example-test/__init__.py # python package marker - must always exist.
700 ./example-test/test_template.jpg # generated topology picture - see next section
701 ./example-test/test_template.dot # Graphviz dot file
702 ./example-test/test_template.py # the topology plus the test
704 ./ospf-topo1 # the ospf topology test
705 ./ospf-topo1/r1 # router 1 configuration files
706 ./ospf-topo1/r1/zebra.conf # zebra configuration file
707 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
708 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
709 # removed other for shortness sake
711 ./lib # shared test/topology functions
712 ./lib/topogen.py # topogen implementation
713 ./lib/topotest.py # topotest implementation
715 Guidelines for creating/editing topotest:
717 - New topologies that don't fit the existing directories should create its own
718 - Always remember to add the ``__init__.py`` to new folders, this makes auto
719 complete engines and pylint happy
720 - Router (Quagga/FRR) specific code should go on topotest.py
721 - Generic/repeated router actions should have an abstraction in
723 - Generic/repeated non-router code should go to topotest.py
724 - pytest related code should go to conftest.py (e.g. specialized asserts)
726 Defining the Topology
727 """""""""""""""""""""
729 The first step to write a new test is to define the topology. This step can be
730 done in many ways, but the recommended is to use Graphviz to generate a drawing
731 of the topology. It allows us to see the topology graphically and to see the
732 names of equipment, links and addresses.
734 Here is an example of Graphviz dot file that generates the template topology
735 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
736 get outdated, please see the linked file)::
758 label="s1\n192.168.0.0/24",
764 label="s2\n192.168.1.0/24",
770 r1 -- s1 [label="eth0\n.1"];
772 r1 -- s2 [label="eth1\n.100"];
773 r2 -- s2 [label="eth0\n.1"];
776 Here is the produced graph:
800 label="s1\n192.168.0.0/24",
806 label="s2\n192.168.1.0/24",
812 r1 -- s1 [label="eth0\n.1"];
814 r1 -- s2 [label="eth1\n.100"];
815 r2 -- s2 [label="eth0\n.1"];
818 Generating / Obtaining Configuration Files
819 """"""""""""""""""""""""""""""""""""""""""
821 In order to get the configuration files or command output for each router, we
822 need to run the topology and execute commands in ``vtysh``. The quickest way to
823 achieve that is writing the topology building code and running the topology.
825 To bootstrap your test topology, do the following steps:
827 - Copy the template test
832 $ touch new-topo/__init__.py
833 $ cp example-test/test_template.py new-topo/test_new_topo.py
835 - Modify the template according to your dot file
837 Here is the template topology described in the previous section in python code:
846 If more specialized topology definitions, or router initialization arguments are
847 required a build function can be used instead of a dictionary:
851 def build_topo(tgen):
855 for routern in range(1, 3):
856 tgen.add_router("r{}".format(routern))
858 # Create a switch with just one router connected to it to simulate a
860 switch = tgen.add_switch("s1")
861 switch.add_link(tgen.gears["r1"])
863 # Create a connection between r1 and r2
864 switch = tgen.add_switch("s2")
865 switch.add_link(tgen.gears["r1"])
866 switch.add_link(tgen.gears["r2"])
870 Topogen allows us to run the topology without running any tests, you can do
871 that using the following example commands:
875 $ # Running your bootstraped topology
876 $ sudo -E pytest -s --topology-only new-topo/test_new_topo.py
877 $ # Running the test_template.py topology
878 $ sudo -E pytest -s --topology-only example-test/test_template.py
879 $ # Running the ospf_topo1.py topology
880 $ sudo -E pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
882 Parameters explanation:
888 Actives input/output capture. If this is not specified a new window will be
889 opened for the interactive CLI, otherwise it will be activated inline.
891 .. option:: --topology-only
893 Don't run any tests, just build the topology.
895 After executing the commands above, you should get the following terminal
900 frr/tests/topotests# sudo -E pytest -s --topology-only ospf_topo1/test_ospf_topo1.py
901 ============================= test session starts ==============================
902 platform linux -- Python 3.9.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
903 rootdir: /home/chopps/w/frr/tests/topotests, configfile: pytest.ini
904 plugins: forked-1.3.0, xdist-2.3.0
910 The last line shows us that we are now using the CLI (Command Line
911 Interface), from here you can call your router ``vtysh`` or even bash.
913 Here's the help text:
921 sh [hosts] <shell-command> :: execute <shell-command> on <host>
922 term [hosts] :: open shell terminals for hosts
923 vtysh [hosts] :: open vtysh terminals for hosts
924 [hosts] <vtysh-command> :: execute vtysh-command on hosts
926 Here are some commands example:
930 unet> sh r1 ping 10.0.3.1
931 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
932 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
933 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
934 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
936 --- 10.0.3.1 ping statistics ---
937 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
938 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
941 Building configuration...
943 Current configuration:
945 frr version 8.1-dev-my-manual-build
946 frr defaults traditional
948 log file /tmp/topotests/ospf_topo1.test_ospf_topo1/r1/zebra.log
953 ------ Host: r1 ------
954 zebra ospfd ospf6d staticd
955 ------- End: r1 ------
956 ------ Host: r2 ------
957 zebra ospfd ospf6d staticd
958 ------- End: r2 ------
959 ------ Host: r3 ------
960 zebra ospfd ospf6d staticd
961 ------- End: r3 ------
962 ------ Host: r4 ------
963 zebra ospfd ospf6d staticd
964 ------- End: r4 ------
966 After you successfully configured your topology, you can obtain the
967 configuration files (per-daemon) using the following commands:
971 unet> sh r3 vtysh -d ospfd
973 Hello, this is FRRouting (version 3.1-devrzalamena-build).
974 Copyright 1996-2005 Kunihiro Ishiguro, et al.
976 r1# show running-config
977 Building configuration...
979 Current configuration:
981 frr version 3.1-devrzalamena-build
982 frr defaults traditional
983 no service integrated-vtysh-config
988 ospf router-id 10.0.255.3
990 redistribute connected
992 network 10.0.3.0/24 area 0
993 network 10.0.10.0/24 area 0
994 network 172.16.0.0/24 area 1
1001 You can also login to the node specified by nsenter using bash, etc.
1002 A pid file for each node will be created in the relevant test dir.
1003 You can run scripts inside the node, or use vtysh's <tab> or <?> feature.
1008 # cd tests/topotests/srv6_locator
1009 # ./test_srv6_locator.py --topology-only
1010 unet> r1 show segment-routing srv6 locator
1012 Name ID Prefix Status
1013 -------------------- ------- ------------------------ -------
1014 loc1 1 2001:db8:1:1::/64 Up
1015 loc2 2 2001:db8:2:2::/64 Up
1018 # nsenter -a -t $(cat /tmp/topotests/srv6_locator.test_srv6_locator/r1.pid) bash --norc
1020 r1# r1 show segment-routing srv6 locator
1022 Name ID Prefix Status
1023 -------------------- ------- ------------------------ -------
1024 loc1 1 2001:db8:1:1::/64 Up
1025 loc2 2 2001:db8:2:2::/64 Up
1030 Test topologies should always be bootstrapped from
1031 :file:`tests/topotests/example_test/test_template.py` because it contains
1032 important boilerplate code that can't be avoided, like:
1038 # For all routers arrange for:
1039 # - starting zebra using config file from <rtrname>/zebra.conf
1040 # - starting ospfd using an empty config file.
1041 for rname, router in router_list.items():
1042 router.load_config(TopoRouter.RD_ZEBRA, "zebra.conf")
1043 router.load_config(TopoRouter.RD_OSPF)
1046 - The topology definition or build function
1055 def build_topo(tgen):
1056 # topology build code
1059 - pytest setup/teardown fixture to start the topology and supply ``tgen``
1065 @pytest.fixture(scope="module")
1067 "Setup/Teardown the environment and provide tgen argument to tests"
1069 tgen = Topogen(topodef, module.__name__)
1071 tgen = Topogen(build_topo, module.__name__)
1075 # Start and configure the router daemons
1078 # Provide tgen as argument to each test function
1081 # Teardown after last test runs
1082 tgen.stop_topology()
1087 - Directory name for a new topotest must not contain hyphen (``-``) characters.
1088 To separate words, use underscores (``_``). For example, ``tests/topotests/bgp_new_example``.
1089 - Test code should always be declared inside functions that begin with the
1090 ``test_`` prefix. Functions beginning with different prefixes will not be run
1092 - Configuration files and long output commands should go into separated files
1093 inside folders named after the equipment.
1094 - Tests must be able to run without any interaction. To make sure your test
1095 conforms with this, run it without the :option:`-s` parameter.
1096 - Use `black <https://github.com/psf/black>`_ code formatter before creating
1097 a pull request. This ensures we have a unified code style.
1098 - Mark test modules with pytest markers depending on the daemons used during the
1099 tests (see :ref:`topotests-markers`)
1100 - Always use IPv4 :rfc:`5737` (``192.0.2.0/24``, ``198.51.100.0/24``,
1101 ``203.0.113.0/24``) and IPv6 :rfc:`3849` (``2001:db8::/32``) ranges reserved
1106 - Keep results in stack variables, so people inspecting code with ``pdb`` can
1107 easily print their values.
1113 assert foobar(router1, router2)
1119 result = foobar(router1, router2)
1122 - Use ``assert`` messages to indicate where the test failed.
1128 for router in router_list:
1130 assert condition, 'Router "{}" condition failed'.format(router.name)
1135 The most effective ways to inspect topology tests are:
1137 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
1138 appear when an assertion fails
1140 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
1142 - Set a breakpoint in the test code with ``pdb``
1148 # Add the pdb import at the beginning of the file
1152 # Add a breakpoint where you think the problem is
1158 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
1159 shell allows us to run many useful operations like:
1161 - Setting breaking point on file/function/conditions (e.g. ``break``,
1163 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
1164 - Running python code
1168 The TopoGear (equipment abstraction class) implements the ``__str__`` method
1169 that allows the user to inspect equipment information.
1171 Example of pdb usage:
1175 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
1176 -> for rnum in range(1, 5):
1178 Documented commands (type help <topic>):
1179 ========================================
1180 EOF bt cont enable jump pp run unt
1181 a c continue exit l q s until
1182 alias cl d h list quit step up
1183 args clear debug help n r tbreak w
1184 b commands disable ignore next restart u whatis
1185 break condition down j p return unalias where
1187 Miscellaneous help topics:
1188 ==========================
1191 Undocumented commands:
1192 ======================
1196 116 title2="Expected output")
1198 118 def test_ospf_convergence():
1199 119 "Test OSPF daemon convergence"
1201 121 -> for rnum in range(1, 5):
1202 122 router = 'r{}'.format(rnum)
1204 124 # Load expected results from the command
1205 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1206 126 expected = open(reffile).read()
1208 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
1209 -> router = 'r{}'.format(rnum)
1211 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
1212 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1217 (Pdb) tgen = get_topogen()
1218 (Pdb) pp tgen.gears[router]
1219 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
1220 (Pdb) pp str(tgen.gears[router])
1221 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
1224 121 for rnum in range(1, 5):
1225 122 router = 'r{}'.format(rnum)
1227 124 # Load expected results from the command
1228 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
1229 126 expected = open(reffile).read()
1231 128 # Run test function until we get an result. Wait at most 60 seconds.
1232 129 test_func = partial(compare_show_ip_ospf, router, expected)
1233 130 result, diff = topotest.run_and_expect(test_func, '',
1234 (Pdb) router1 = tgen.gears[router]
1235 (Pdb) router1.vtysh_cmd('show ip ospf route')
1236 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
1240 To enable more debug messages in other Topogen subsystems, more
1241 logging messages can be displayed by modifying the test configuration file
1247 # Change the default verbosity line from 'info'...
1252 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
1253 To learn/remember common code snippets see :ref:`topotests-snippets`.
1255 Before creating a new topology, make sure that there isn't one already that
1256 does what you need. If nothing is similar, then you may create a new topology,
1257 preferably, using the newest template
1258 (:file:`tests/topotests/example-test/test_template.py`).
1260 .. include:: topotests-markers.rst
1262 .. include:: topotests-snippets.rst
1267 All the configs and scripts are licensed under a ISC-style license. See Python
1268 scripts for details.