6 Topotests is a suite of topology tests for FRR built on top of Mininet.
11 Only tested with Ubuntu 16.04 and Ubuntu 18.04 (which uses Mininet 2.2.x).
13 Instructions are the same for all setups (i.e. ExaBGP is only used for BGP
16 Installing Mininet Infrastructure
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 apt-get install mininet
22 apt-get install python-pip
23 apt-get install iproute
25 pip install "pytest<5"
26 pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet
28 useradd -d /var/run/exabgp/ -s /bin/false exabgp
33 Optional, will give better output.
38 disable apport (which move core files)
40 Set ``enabled=0`` in ``/etc/default/apport``.
42 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
44 #<domain> <type> <item> <value>
46 root soft core unlimited
48 root hard core unlimited
50 Reboot for options to take effect.
55 FRR needs to be installed separately. It is assume to be configured like the
56 standard Ubuntu Packages:
58 - Binaries in :file:`/usr/lib/frr`
59 - State Directory :file:`/var/run/frr`
60 - Running under user ``frr``, group ``frr``
61 - vtygroup: ``frrvty``
62 - config directory: :file:`/etc/frr`
63 - For FRR Packages, install the dbg package as well for coredump decoding
65 No FRR config needs to be done and no FRR daemons should be run ahead of the
66 test. They are all started as part of the test.
71 If you prefer to manually build FRR, then use the following suggested config:
77 --localstatedir=/var/run/frr \
78 --sbindir=/usr/lib/frr \
79 --sysconfdir=/etc/frr \
83 --enable-multipath=64 \
86 --enable-vty-group=frrvty \
87 --with-pkg-extra-version=-my-manual-build
89 And create ``frr`` user and ``frrvty`` group as follows:
93 addgroup --system --gid 92 frr
94 addgroup --system --gid 85 frrvty
95 adduser --system --ingroup frr --home /var/run/frr/ \
96 --gecos "FRRouting suite" --shell /bin/false frr
102 Execute all tests with output to console
103 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
107 py.test -s -v --tb=no
109 The above command must be executed from inside the topotests directory.
111 All test\_\* scripts in subdirectories are detected and executed (unless
112 disabled in ``pytest.ini`` file).
114 ``--tb=no`` disables the python traceback which might be irrelevant unless the
115 test script itself is debugged.
125 For example, and assuming you are inside the frr directory:
129 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
130 ./test_bgp_l3vpn_to_bgp_vrf.py
132 For further options, refer to pytest documentation.
134 Test will set exit code which can be used with ``git bisect``.
136 For the simulated topology, see the description in the python file.
138 If you need to clear the mininet setup between tests (if it isn't cleanly
139 shutdown), then use the ``mn -c`` command to clean up the environment.
141 StdErr log from daemos after exit
142 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
144 To enable the reporting of any messages seen on StdErr after the daemons exit,
145 the following env variable can be set::
147 export TOPOTESTS_CHECK_STDERR=Yes
149 (The value doesn't matter at this time. The check is whether the env
150 variable exists or not.) There is no pass/fail on this reporting; the
151 Output will be reported to the console.
153 Collect Memory Leak Information
154 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
156 FRR processes can report unfreed memory allocations upon exit. To
157 enable the reporting of memory leaks, define an environment variable
158 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
160 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
162 This will enable the check and output to console and the writing of
163 the information to files with the given prefix (followed by testname),
164 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
167 Running Topotests with AddressSanitizer
168 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
170 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
171 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
172 information on AddressSanitizer, see
173 https://github.com/google/sanitizers/wiki/AddressSanitizer.
175 The checks are done automatically in the library call of ``checkRouterRunning``
176 (ie at beginning of tests when there is a check for all daemons running). No
177 changes or extra configuration for topotests is required beside compiling the
178 suite with AddressSanitizer enabled.
180 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
181 If found, then this is added with context (calling test) to
182 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
184 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
185 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
186 for ``master`` branch:
190 git clone https://github.com/FRRouting/frr.git
194 --enable-address-sanitizer \
195 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
196 --localstatedir=/var/run/frr \
197 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
198 --enable-exampledir=/usr/lib/frr/examples \
199 --with-moduledir=/usr/lib/frr/modules \
200 --enable-multipath=0 --enable-rtadv \
201 --enable-tcp-zebra --enable-fpm --enable-pimd \
205 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
206 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
208 and create ``frr`` user and ``frrvty`` group as shown above.
210 .. _topotests_docker:
212 Running Tests with Docker
213 -------------------------
215 There is a Docker image which allows to run topotests.
220 If you have Docker installed, you can run the topotests in Docker. The easiest
221 way to do this, is to use the make targets from this repository.
223 Your current user needs to have access to the Docker daemon. Alternatively you
224 can run these commands as root.
230 This command will pull the most recent topotests image from Dockerhub, compile
231 FRR inside of it, and run the topotests.
236 Internally, the topotests make target uses a shell script to pull the image and
237 spawn the Docker container.
239 There are several environment variables which can be used to modify the
240 behavior of the script, these can be listed by calling it with ``-h``:
244 ./tests/topotests/docker/frr-topotests.sh -h
246 For example, a volume is used to cache build artifacts between multiple runs of
247 the image. If you need to force a complete recompile, you can set
252 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
254 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
255 arguments and the first one starts with ``/`` or ``./``, they will replace the
256 call to pytest. If the appended arguments do not match this patttern, they will
257 be provided to pytest as arguments. So, to run a specific test with more
262 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
264 And to compile FRR but drop into a shell instead of running pytest:
268 ./tests/topotests/docker/frr-topotests.sh /bin/bash
273 The Docker image just includes all the components to run the topotests, but not
274 the topotests themselves. So if you just want to write tests and don't want to
275 make changes to the environment provided by the Docker image. You don't need to
276 build your own Docker image if you do not want to.
278 When developing new tests, there is one caveat though: The startup script of
279 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
280 pollution of the container with build artefacts from the host. This will also
281 result in your newly written tests being unavailable in the container unless at
282 least added to the index with ``git-add``.
284 If you do want to test changes to the Docker image, you can locally build the
285 image and run the tests without pulling from the registry using the following
291 TOPOTEST_PULL=0 make topotests
294 .. _topotests-guidelines:
302 To run the whole suite of tests the following commands must be executed at the
303 top level directory of topotest:
307 $ # Change to the top level directory of topotests.
308 $ cd path/to/topotests
309 $ # Tests must be run as root, since Mininet requires it.
312 In order to run a specific test, you can use the following command:
316 $ # running a specific topology
317 $ sudo pytest ospf-topo1/
318 $ # or inside the test folder
320 $ sudo pytest # to run all tests inside the directory
321 $ sudo pytest test_ospf_topo1.py # to run a specific test
322 $ # or outside the test folder
324 $ sudo pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
326 The output of the tested daemons will be available at the temporary folder of
331 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
333 zebra.err # zebra stderr output
334 zebra.log # zebra log file
335 zebra.out # zebra stdout output
338 You can also run memory leak tests to get reports:
342 $ # Set the environment variable to apply to a specific test...
343 $ sudo env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
344 $ # ...or apply to all tests adding this line to the configuration file
345 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
346 $ # You can also use your editor
348 $ # After running tests you should see your files:
349 $ ls /tmp/memleak_report_*
350 memleak_report_test_ospf_topo1.txt
355 This section will guide you in all recommended steps to produce a standard
358 This is the recommended test writing routine:
360 - Write a topology (Graphviz recommended)
361 - Obtain configuration files
362 - Write the test itself
363 - Format the new code using `black <https://github.com/psf/black>`_
364 - Create a Pull Request
368 BGP tests MUST use generous convergence timeouts - you must ensure
369 that any test involving BGP uses a convergence timeout of at least
372 Topotest File Hierarchy
373 """""""""""""""""""""""
375 Before starting to write any tests one must know the file hierarchy. The
376 repository hierarchy looks like this:
380 $ cd path/to/topotest
383 ./README.md # repository read me
384 ./GUIDELINES.md # this file
385 ./conftest.py # test hooks - pytest related functions
386 ./example-test # example test folder
387 ./example-test/__init__.py # python package marker - must always exist.
388 ./example-test/test_template.jpg # generated topology picture - see next section
389 ./example-test/test_template.dot # Graphviz dot file
390 ./example-test/test_template.py # the topology plus the test
392 ./ospf-topo1 # the ospf topology test
393 ./ospf-topo1/r1 # router 1 configuration files
394 ./ospf-topo1/r1/zebra.conf # zebra configuration file
395 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
396 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
397 # removed other for shortness sake
399 ./lib # shared test/topology functions
400 ./lib/topogen.py # topogen implementation
401 ./lib/topotest.py # topotest implementation
403 Guidelines for creating/editing topotest:
405 - New topologies that don't fit the existing directories should create its own
406 - Always remember to add the ``__init__.py`` to new folders, this makes auto
407 complete engines and pylint happy
408 - Router (Quagga/FRR) specific code should go on topotest.py
409 - Generic/repeated router actions should have an abstraction in
411 - Generic/repeated non-router code should go to topotest.py
412 - pytest related code should go to conftest.py (e.g. specialized asserts)
414 Defining the Topology
415 """""""""""""""""""""
417 The first step to write a new test is to define the topology. This step can be
418 done in many ways, but the recommended is to use Graphviz to generate a drawing
419 of the topology. It allows us to see the topology graphically and to see the
420 names of equipment, links and addresses.
422 Here is an example of Graphviz dot file that generates the template topology
423 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
424 get outdated, please see the linked file)::
446 label="s1\n192.168.0.0/24",
452 label="s2\n192.168.1.0/24",
458 r1 -- s1 [label="eth0\n.1"];
460 r1 -- s2 [label="eth1\n.100"];
461 r2 -- s2 [label="eth0\n.1"];
464 Here is the produced graph:
488 label="s1\n192.168.0.0/24",
494 label="s2\n192.168.1.0/24",
500 r1 -- s1 [label="eth0\n.1"];
502 r1 -- s2 [label="eth1\n.100"];
503 r2 -- s2 [label="eth0\n.1"];
506 Generating / Obtaining Configuration Files
507 """"""""""""""""""""""""""""""""""""""""""
509 In order to get the configuration files or command output for each router, we
510 need to run the topology and execute commands in ``vtysh``. The quickest way to
511 achieve that is writing the topology building code and running the topology.
513 To bootstrap your test topology, do the following steps:
515 - Copy the template test
520 $ touch new-topo/__init__.py
521 $ cp example-test/test_template.py new-topo/test_new_topo.py
523 - Modify the template according to your dot file
525 Here is the template topology described in the previous section in python code:
529 class TemplateTopo(Topo):
530 "Test topology builder"
531 def build(self, *_args, **_opts):
533 tgen = get_topogen(self)
536 for routern in range(1, 3):
537 tgen.add_router('r{}'.format(routern))
539 # Create a switch with just one router connected to it to simulate a
541 switch = tgen.add_switch('s1')
542 switch.add_link(tgen.gears['r1'])
544 # Create a connection between r1 and r2
545 switch = tgen.add_switch('s2')
546 switch.add_link(tgen.gears['r1'])
547 switch.add_link(tgen.gears['r2'])
551 Topogen allows us to run the topology without running any tests, you can do
552 that using the following example commands:
556 $ # Running your bootstraped topology
557 $ sudo pytest -s --topology-only new-topo/test_new_topo.py
558 $ # Running the test_template.py topology
559 $ sudo pytest -s --topology-only example-test/test_template.py
560 $ # Running the ospf_topo1.py topology
561 $ sudo pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
563 Parameters explanation:
569 Actives input/output capture. This is required by mininet in order to show
570 the interactive shell.
572 .. option:: --topology-only
574 Don't run any tests, just build the topology.
576 After executing the commands above, you should get the following terminal
581 === test session starts ===
582 platform linux2 -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
583 rootdir: /media/sf_src/topotests, inifile: pytest.ini
586 ospf-topo1/test_ospf_topo1.py *** Starting controller
588 *** Starting 6 switches
589 switch1 switch2 switch3 switch4 switch5 switch6 ...
590 r2: frr zebra started
591 r2: frr ospfd started
592 r3: frr zebra started
593 r3: frr ospfd started
594 r1: frr zebra started
595 r1: frr ospfd started
596 r4: frr zebra started
597 r4: frr ospfd started
601 The last line shows us that we are now using the Mininet CLI (Command Line
602 Interface), from here you can call your router ``vtysh`` or even bash.
604 Here are some commands example:
608 mininet> r1 ping 10.0.3.1
609 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
610 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
611 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
612 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
614 --- 10.0.3.1 ping statistics ---
615 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
616 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
620 mininet> r1 ping 10.0.3.3
621 PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
622 64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.87 ms
623 64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.080 ms
624 64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=0.091 ms
626 --- 10.0.3.3 ping statistics ---
627 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
628 rtt min/avg/max/mdev = 0.080/1.014/2.872/1.313 ms
634 Hello, this is FRRouting (version 3.1-devrzalamena-build).
635 Copyright 1996-2005 Kunihiro Ishiguro, et al.
637 frr-1# show running-config
638 Building configuration...
640 Current configuration:
642 frr version 3.1-devrzalamena-build
643 frr defaults traditional
645 no service integrated-vtysh-config
652 ip address 10.0.3.1/24
655 ip address 10.0.10.1/24
658 ip address 172.16.0.2/24
661 ospf router-id 10.0.255.3
663 redistribute connected
665 network 10.0.3.0/24 area 0
666 network 10.0.10.0/24 area 0
667 network 172.16.0.0/24 area 1
674 After you successfully configured your topology, you can obtain the
675 configuration files (per-daemon) using the following commands:
679 mininet> r3 vtysh -d ospfd
681 Hello, this is FRRouting (version 3.1-devrzalamena-build).
682 Copyright 1996-2005 Kunihiro Ishiguro, et al.
684 frr-1# show running-config
685 Building configuration...
687 Current configuration:
689 frr version 3.1-devrzalamena-build
690 frr defaults traditional
691 no service integrated-vtysh-config
696 ospf router-id 10.0.255.3
698 redistribute connected
700 network 10.0.3.0/24 area 0
701 network 10.0.10.0/24 area 0
702 network 172.16.0.0/24 area 1
712 Test topologies should always be bootstrapped from
713 :file:`tests/topotests/example-test/test_template.py` because it contains
714 important boilerplate code that can't be avoided, like:
716 - imports: os, sys, pytest, topotest/topogen and mininet topology class
717 - The global variable CWD (Current Working directory): which is most likely
718 going to be used to reference the routers configuration file location
724 # For all registered routers, load the zebra configuration file
725 for rname, router in router_list.items():
728 os.path.join(CWD, '{}/zebra.conf'.format(rname))
730 # os.path.join() joins the CWD string with arguments adding the necessary
731 # slashes ('/'). Arguments must not begin with '/'.
733 - The topology class that inherits from Mininet Topo class:
737 class TemplateTopo(Topo):
738 def build(self, *_args, **_opts):
739 tgen = get_topogen(self)
740 # topology build code
742 - pytest ``setup_module()`` and ``teardown_module()`` to start the topology
746 def setup_module(_m):
747 tgen = Topogen(TemplateTopo)
748 tgen.start_topology('debug')
750 def teardown_module(_m):
754 - ``__main__`` initialization code (to support running the script directly)
758 if __name__ == '__main__':
759 sys.exit(pytest.main(["-s"]))
763 - Test code should always be declared inside functions that begin with the
764 ``test_`` prefix. Functions beginning with different prefixes will not be run
766 - Configuration files and long output commands should go into separated files
767 inside folders named after the equipment.
768 - Tests must be able to run without any interaction. To make sure your test
769 conforms with this, run it without the :option:`-s` parameter.
770 - Use `black <https://github.com/psf/black>`_ code formatter before creating
771 a pull request. This ensures we have a unified code style.
775 - Keep results in stack variables, so people inspecting code with ``pdb`` can
776 easily print their values.
782 assert foobar(router1, router2)
788 result = foobar(router1, router2)
791 - Use ``assert`` messages to indicate where the test failed.
797 for router in router_list:
799 assert condition, 'Router "{}" condition failed'.format(router.name)
804 The most effective ways to inspect topology tests are:
806 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
807 appear when an assertion fails
809 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
811 - Set a breakpoint in the test code with ``pdb``
817 # Add the pdb import at the beginning of the file
821 # Add a breakpoint where you think the problem is
827 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
828 shell allows us to run many useful operations like:
830 - Setting breaking point on file/function/conditions (e.g. ``break``,
832 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
833 - Running python code
837 The TopoGear (equipment abstraction class) implements the ``__str__`` method
838 that allows the user to inspect equipment information.
840 Example of pdb usage:
844 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
845 -> for rnum in range(1, 5):
847 Documented commands (type help <topic>):
848 ========================================
849 EOF bt cont enable jump pp run unt
850 a c continue exit l q s until
851 alias cl d h list quit step up
852 args clear debug help n r tbreak w
853 b commands disable ignore next restart u whatis
854 break condition down j p return unalias where
856 Miscellaneous help topics:
857 ==========================
860 Undocumented commands:
861 ======================
865 116 title2="Expected output")
867 118 def test_ospf_convergence():
868 119 "Test OSPF daemon convergence"
870 121 -> for rnum in range(1, 5):
871 122 router = 'r{}'.format(rnum)
873 124 # Load expected results from the command
874 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
875 126 expected = open(reffile).read()
877 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
878 -> router = 'r{}'.format(rnum)
880 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
881 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
886 (Pdb) tgen = get_topogen()
887 (Pdb) pp tgen.gears[router]
888 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
889 (Pdb) pp str(tgen.gears[router])
890 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
893 121 for rnum in range(1, 5):
894 122 router = 'r{}'.format(rnum)
896 124 # Load expected results from the command
897 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
898 126 expected = open(reffile).read()
900 128 # Run test function until we get an result. Wait at most 60 seconds.
901 129 test_func = partial(compare_show_ip_ospf, router, expected)
902 130 result, diff = topotest.run_and_expect(test_func, '',
903 (Pdb) router1 = tgen.gears[router]
904 (Pdb) router1.vtysh_cmd('show ip ospf route')
905 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
906 (Pdb) tgen.mininet_cli()
910 To enable more debug messages in other Topogen subsystems (like Mininet), more
911 logging messages can be displayed by modifying the test configuration file
917 # Change the default verbosity line from 'info'...
922 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
923 To learn/remember common code snippets see :ref:`topotests-snippets`.
925 Before creating a new topology, make sure that there isn't one already that
926 does what you need. If nothing is similar, then you may create a new topology,
927 preferably, using the newest template
928 (:file:`tests/topotests/example-test/test_template.py`).
930 .. include:: topotests-snippets.rst
935 All the configs and scripts are licensed under a ISC-style license. See Python