6 Topotests is a suite of topology tests for FRR built on top of Mininet.
11 Only tested with Ubuntu 16.04 and Ubuntu 18.04 (which uses Mininet 2.2.x).
13 Instructions are the same for all setups (i.e. ExaBGP is only used for BGP
16 Installing Mininet Infrastructure
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 apt-get install mininet
22 apt-get install python-pip
23 apt-get install iproute
25 pip install "pytest<5"
26 pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet
28 useradd -d /var/run/exabgp/ -s /bin/false exabgp
33 Optional, will give better output.
38 disable apport (which move core files)
40 Set ``enabled=0`` in ``/etc/default/apport``.
42 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
44 #<domain> <type> <item> <value>
46 root soft core unlimited
48 root hard core unlimited
50 Reboot for options to take effect.
55 FRR needs to be installed separately. It is assume to be configured like the
56 standard Ubuntu Packages:
58 - Binaries in :file:`/usr/lib/frr`
59 - State Directory :file:`/var/run/frr`
60 - Running under user ``frr``, group ``frr``
61 - vtygroup: ``frrvty``
62 - config directory: :file:`/etc/frr`
63 - For FRR Packages, install the dbg package as well for coredump decoding
65 No FRR config needs to be done and no FRR daemons should be run ahead of the
66 test. They are all started as part of the test.
71 If you prefer to manually build FRR, then use the following suggested config:
77 --localstatedir=/var/run/frr \
78 --sbindir=/usr/lib/frr \
79 --sysconfdir=/etc/frr \
83 --enable-multipath=64 \
86 --enable-vty-group=frrvty \
87 --with-pkg-extra-version=-my-manual-build
89 And create ``frr`` user and ``frrvty`` group as follows:
93 addgroup --system --gid 92 frr
94 addgroup --system --gid 85 frrvty
95 adduser --system --ingroup frr --home /var/run/frr/ \
96 --gecos "FRRouting suite" --shell /bin/false frr
102 Execute all tests with output to console
103 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
107 py.test -s -v --tb=no
109 The above command must be executed from inside the topotests directory.
111 All test\_\* scripts in subdirectories are detected and executed (unless
112 disabled in ``pytest.ini`` file).
114 ``--tb=no`` disables the python traceback which might be irrelevant unless the
115 test script itself is debugged.
125 For example, and assuming you are inside the frr directory:
129 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
130 ./test_bgp_l3vpn_to_bgp_vrf.py
132 For further options, refer to pytest documentation.
134 Test will set exit code which can be used with ``git bisect``.
136 For the simulated topology, see the description in the python file.
138 If you need to clear the mininet setup between tests (if it isn't cleanly
139 shutdown), then use the ``mn -c`` command to clean up the environment.
141 StdErr log from daemos after exit
142 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
144 To enable the reporting of any messages seen on StdErr after the daemons exit,
145 the following env variable can be set::
147 export TOPOTESTS_CHECK_STDERR=Yes
149 (The value doesn't matter at this time. The check is whether the env
150 variable exists or not.) There is no pass/fail on this reporting; the
151 Output will be reported to the console.
153 Collect Memory Leak Information
154 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
156 FRR processes can report unfreed memory allocations upon exit. To
157 enable the reporting of memory leaks, define an environment variable
158 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
160 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
162 This will enable the check and output to console and the writing of
163 the information to files with the given prefix (followed by testname),
164 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
167 Running Topotests with AddressSanitizer
168 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
170 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
171 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
172 information on AddressSanitizer, see
173 https://github.com/google/sanitizers/wiki/AddressSanitizer.
175 The checks are done automatically in the library call of ``checkRouterRunning``
176 (ie at beginning of tests when there is a check for all daemons running). No
177 changes or extra configuration for topotests is required beside compiling the
178 suite with AddressSanitizer enabled.
180 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
181 If found, then this is added with context (calling test) to
182 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
184 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
185 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
186 for ``master`` branch:
190 git clone https://github.com/FRRouting/frr.git
194 --enable-address-sanitizer \
195 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
196 --localstatedir=/var/run/frr \
197 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
198 --enable-exampledir=/usr/lib/frr/examples \
199 --with-moduledir=/usr/lib/frr/modules \
200 --enable-multipath=0 --enable-rtadv \
201 --enable-tcp-zebra --enable-fpm --enable-pimd \
205 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
206 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
208 and create ``frr`` user and ``frrvty`` group as shown above.
210 .. _topotests_docker:
212 Running Tests with Docker
213 -------------------------
215 There is a Docker image which allows to run topotests.
220 If you have Docker installed, you can run the topotests in Docker. The easiest
221 way to do this, is to use the make targets from this repository.
223 Your current user needs to have access to the Docker daemon. Alternatively you
224 can run these commands as root.
230 This command will pull the most recent topotests image from Dockerhub, compile
231 FRR inside of it, and run the topotests.
236 Internally, the topotests make target uses a shell script to pull the image and
237 spawn the Docker container.
239 There are several environment variables which can be used to modify the
240 behavior of the script, these can be listed by calling it with ``-h``:
244 ./tests/topotests/docker/frr-topotests.sh -h
246 For example, a volume is used to cache build artifacts between multiple runs of
247 the image. If you need to force a complete recompile, you can set
252 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
254 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
255 arguments and the first one starts with ``/`` or ``./``, they will replace the
256 call to pytest. If the appended arguments do not match this patttern, they will
257 be provided to pytest as arguments. So, to run a specific test with more
262 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
264 And to compile FRR but drop into a shell instead of running pytest:
268 ./tests/topotests/docker/frr-topotests.sh /bin/bash
273 The Docker image just includes all the components to run the topotests, but not
274 the topotests themselves. So if you just want to write tests and don't want to
275 make changes to the environment provided by the Docker image. You don't need to
276 build your own Docker image if you do not want to.
278 When developing new tests, there is one caveat though: The startup script of
279 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
280 pollution of the container with build artefacts from the host. This will also
281 result in your newly written tests being unavailable in the container unless at
282 least added to the index with ``git-add``.
284 If you do want to test changes to the Docker image, you can locally build the
285 image and run the tests without pulling from the registry using the following
291 TOPOTEST_PULL=0 make topotests
294 .. _topotests-guidelines:
302 To run the whole suite of tests the following commands must be executed at the
303 top level directory of topotest:
307 $ # Change to the top level directory of topotests.
308 $ cd path/to/topotests
309 $ # Tests must be run as root, since Mininet requires it.
312 In order to run a specific test, you can use the following command:
316 $ # running a specific topology
317 $ sudo pytest ospf-topo1/
318 $ # or inside the test folder
320 $ sudo pytest # to run all tests inside the directory
321 $ sudo pytest test_ospf_topo1.py # to run a specific test
322 $ # or outside the test folder
324 $ sudo pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
326 The output of the tested daemons will be available at the temporary folder of
331 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
333 zebra.err # zebra stderr output
334 zebra.log # zebra log file
335 zebra.out # zebra stdout output
338 You can also run memory leak tests to get reports:
342 $ # Set the environment variable to apply to a specific test...
343 $ sudo env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
344 $ # ...or apply to all tests adding this line to the configuration file
345 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
346 $ # You can also use your editor
348 $ # After running tests you should see your files:
349 $ ls /tmp/memleak_report_*
350 memleak_report_test_ospf_topo1.txt
355 This section will guide you in all recommended steps to produce a standard
358 This is the recommended test writing routine:
360 - Write a topology (Graphviz recommended)
361 - Obtain configuration files
362 - Write the test itself
363 - Format the new code using `black <https://github.com/psf/black>`_
364 - Create a Pull Request
366 Topotest File Hierarchy
367 """""""""""""""""""""""
369 Before starting to write any tests one must know the file hierarchy. The
370 repository hierarchy looks like this:
374 $ cd path/to/topotest
377 ./README.md # repository read me
378 ./GUIDELINES.md # this file
379 ./conftest.py # test hooks - pytest related functions
380 ./example-test # example test folder
381 ./example-test/__init__.py # python package marker - must always exist.
382 ./example-test/test_template.jpg # generated topology picture - see next section
383 ./example-test/test_template.dot # Graphviz dot file
384 ./example-test/test_template.py # the topology plus the test
386 ./ospf-topo1 # the ospf topology test
387 ./ospf-topo1/r1 # router 1 configuration files
388 ./ospf-topo1/r1/zebra.conf # zebra configuration file
389 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
390 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
391 # removed other for shortness sake
393 ./lib # shared test/topology functions
394 ./lib/topogen.py # topogen implementation
395 ./lib/topotest.py # topotest implementation
397 Guidelines for creating/editing topotest:
399 - New topologies that don't fit the existing directories should create its own
400 - Always remember to add the ``__init__.py`` to new folders, this makes auto
401 complete engines and pylint happy
402 - Router (Quagga/FRR) specific code should go on topotest.py
403 - Generic/repeated router actions should have an abstraction in
405 - Generic/repeated non-router code should go to topotest.py
406 - pytest related code should go to conftest.py (e.g. specialized asserts)
408 Defining the Topology
409 """""""""""""""""""""
411 The first step to write a new test is to define the topology. This step can be
412 done in many ways, but the recommended is to use Graphviz to generate a drawing
413 of the topology. It allows us to see the topology graphically and to see the
414 names of equipment, links and addresses.
416 Here is an example of Graphviz dot file that generates the template topology
417 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
418 get outdated, please see the linked file)::
440 label="s1\n192.168.0.0/24",
446 label="s2\n192.168.1.0/24",
452 r1 -- s1 [label="eth0\n.1"];
454 r1 -- s2 [label="eth1\n.100"];
455 r2 -- s2 [label="eth0\n.1"];
458 Here is the produced graph:
482 label="s1\n192.168.0.0/24",
488 label="s2\n192.168.1.0/24",
494 r1 -- s1 [label="eth0\n.1"];
496 r1 -- s2 [label="eth1\n.100"];
497 r2 -- s2 [label="eth0\n.1"];
500 Generating / Obtaining Configuration Files
501 """"""""""""""""""""""""""""""""""""""""""
503 In order to get the configuration files or command output for each router, we
504 need to run the topology and execute commands in ``vtysh``. The quickest way to
505 achieve that is writing the topology building code and running the topology.
507 To bootstrap your test topology, do the following steps:
509 - Copy the template test
514 $ touch new-topo/__init__.py
515 $ cp example-test/test_template.py new-topo/test_new_topo.py
517 - Modify the template according to your dot file
519 Here is the template topology described in the previous section in python code:
523 class TemplateTopo(Topo):
524 "Test topology builder"
525 def build(self, *_args, **_opts):
527 tgen = get_topogen(self)
530 for routern in range(1, 3):
531 tgen.add_router('r{}'.format(routern))
533 # Create a switch with just one router connected to it to simulate a
535 switch = tgen.add_switch('s1')
536 switch.add_link(tgen.gears['r1'])
538 # Create a connection between r1 and r2
539 switch = tgen.add_switch('s2')
540 switch.add_link(tgen.gears['r1'])
541 switch.add_link(tgen.gears['r2'])
545 Topogen allows us to run the topology without running any tests, you can do
546 that using the following example commands:
550 $ # Running your bootstraped topology
551 $ sudo pytest -s --topology-only new-topo/test_new_topo.py
552 $ # Running the test_template.py topology
553 $ sudo pytest -s --topology-only example-test/test_template.py
554 $ # Running the ospf_topo1.py topology
555 $ sudo pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
557 Parameters explanation:
563 Actives input/output capture. This is required by mininet in order to show
564 the interactive shell.
566 .. option:: --topology-only
568 Don't run any tests, just build the topology.
570 After executing the commands above, you should get the following terminal
575 === test session starts ===
576 platform linux2 -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
577 rootdir: /media/sf_src/topotests, inifile: pytest.ini
580 ospf-topo1/test_ospf_topo1.py *** Starting controller
582 *** Starting 6 switches
583 switch1 switch2 switch3 switch4 switch5 switch6 ...
584 r2: frr zebra started
585 r2: frr ospfd started
586 r3: frr zebra started
587 r3: frr ospfd started
588 r1: frr zebra started
589 r1: frr ospfd started
590 r4: frr zebra started
591 r4: frr ospfd started
595 The last line shows us that we are now using the Mininet CLI (Command Line
596 Interface), from here you can call your router ``vtysh`` or even bash.
598 Here are some commands example:
602 mininet> r1 ping 10.0.3.1
603 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
604 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
605 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
606 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
608 --- 10.0.3.1 ping statistics ---
609 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
610 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
614 mininet> r1 ping 10.0.3.3
615 PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
616 64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.87 ms
617 64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.080 ms
618 64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=0.091 ms
620 --- 10.0.3.3 ping statistics ---
621 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
622 rtt min/avg/max/mdev = 0.080/1.014/2.872/1.313 ms
628 Hello, this is FRRouting (version 3.1-devrzalamena-build).
629 Copyright 1996-2005 Kunihiro Ishiguro, et al.
631 frr-1# show running-config
632 Building configuration...
634 Current configuration:
636 frr version 3.1-devrzalamena-build
637 frr defaults traditional
639 no service integrated-vtysh-config
646 ip address 10.0.3.1/24
649 ip address 10.0.10.1/24
652 ip address 172.16.0.2/24
655 ospf router-id 10.0.255.3
657 redistribute connected
659 network 10.0.3.0/24 area 0
660 network 10.0.10.0/24 area 0
661 network 172.16.0.0/24 area 1
668 After you successfully configured your topology, you can obtain the
669 configuration files (per-daemon) using the following commands:
673 mininet> r3 vtysh -d ospfd
675 Hello, this is FRRouting (version 3.1-devrzalamena-build).
676 Copyright 1996-2005 Kunihiro Ishiguro, et al.
678 frr-1# show running-config
679 Building configuration...
681 Current configuration:
683 frr version 3.1-devrzalamena-build
684 frr defaults traditional
685 no service integrated-vtysh-config
690 ospf router-id 10.0.255.3
692 redistribute connected
694 network 10.0.3.0/24 area 0
695 network 10.0.10.0/24 area 0
696 network 172.16.0.0/24 area 1
706 Test topologies should always be bootstrapped from
707 :file:`tests/topotests/example-test/test_template.py` because it contains
708 important boilerplate code that can't be avoided, like:
710 - imports: os, sys, pytest, topotest/topogen and mininet topology class
711 - The global variable CWD (Current Working directory): which is most likely
712 going to be used to reference the routers configuration file location
718 # For all registered routers, load the zebra configuration file
719 for rname, router in router_list.iteritems():
722 os.path.join(CWD, '{}/zebra.conf'.format(rname))
724 # os.path.join() joins the CWD string with arguments adding the necessary
725 # slashes ('/'). Arguments must not begin with '/'.
727 - The topology class that inherits from Mininet Topo class:
731 class TemplateTopo(Topo):
732 def build(self, *_args, **_opts):
733 tgen = get_topogen(self)
734 # topology build code
736 - pytest ``setup_module()`` and ``teardown_module()`` to start the topology
740 def setup_module(_m):
741 tgen = Topogen(TemplateTopo)
742 tgen.start_topology('debug')
744 def teardown_module(_m):
748 - ``__main__`` initialization code (to support running the script directly)
752 if __name__ == '__main__':
753 sys.exit(pytest.main(["-s"]))
757 - Test code should always be declared inside functions that begin with the
758 ``test_`` prefix. Functions beginning with different prefixes will not be run
760 - Configuration files and long output commands should go into separated files
761 inside folders named after the equipment.
762 - Tests must be able to run without any interaction. To make sure your test
763 conforms with this, run it without the :option:`-s` parameter.
764 - Use `black <https://github.com/psf/black>`_ code formatter before creating
765 a pull request. This ensures we have a unified code style.
769 - Keep results in stack variables, so people inspecting code with ``pdb`` can
770 easily print their values.
776 assert foobar(router1, router2)
782 result = foobar(router1, router2)
785 - Use ``assert`` messages to indicate where the test failed.
791 for router in router_list:
793 assert condition, 'Router "{}" condition failed'.format(router.name)
798 The most effective ways to inspect topology tests are:
800 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
801 appear when an assertion fails
803 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
805 - Set a breakpoint in the test code with ``pdb``
811 # Add the pdb import at the beginning of the file
815 # Add a breakpoint where you think the problem is
821 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
822 shell allows us to run many useful operations like:
824 - Setting breaking point on file/function/conditions (e.g. ``break``,
826 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
827 - Running python code
831 The TopoGear (equipment abstraction class) implements the ``__str__`` method
832 that allows the user to inspect equipment information.
834 Example of pdb usage:
838 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
839 -> for rnum in range(1, 5):
841 Documented commands (type help <topic>):
842 ========================================
843 EOF bt cont enable jump pp run unt
844 a c continue exit l q s until
845 alias cl d h list quit step up
846 args clear debug help n r tbreak w
847 b commands disable ignore next restart u whatis
848 break condition down j p return unalias where
850 Miscellaneous help topics:
851 ==========================
854 Undocumented commands:
855 ======================
859 116 title2="Expected output")
861 118 def test_ospf_convergence():
862 119 "Test OSPF daemon convergence"
864 121 -> for rnum in range(1, 5):
865 122 router = 'r{}'.format(rnum)
867 124 # Load expected results from the command
868 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
869 126 expected = open(reffile).read()
871 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
872 -> router = 'r{}'.format(rnum)
874 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
875 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
880 (Pdb) tgen = get_topogen()
881 (Pdb) pp tgen.gears[router]
882 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
883 (Pdb) pp str(tgen.gears[router])
884 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
887 121 for rnum in range(1, 5):
888 122 router = 'r{}'.format(rnum)
890 124 # Load expected results from the command
891 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
892 126 expected = open(reffile).read()
894 128 # Run test function until we get an result. Wait at most 60 seconds.
895 129 test_func = partial(compare_show_ip_ospf, router, expected)
896 130 result, diff = topotest.run_and_expect(test_func, '',
897 (Pdb) router1 = tgen.gears[router]
898 (Pdb) router1.vtysh_cmd('show ip ospf route')
899 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
900 (Pdb) tgen.mininet_cli()
904 To enable more debug messages in other Topogen subsystems (like Mininet), more
905 logging messages can be displayed by modifying the test configuration file
911 # Change the default verbosity line from 'info'...
916 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
917 To learn/remember common code snippets see :ref:`topotests-snippets`.
919 Before creating a new topology, make sure that there isn't one already that
920 does what you need. If nothing is similar, then you may create a new topology,
921 preferably, using the newest template
922 (:file:`tests/topotests/example-test/test_template.py`).
924 .. include:: topotests-snippets.rst
929 All the configs and scripts are licensed under a ISC-style license. See Python