6 Topotests is a suite of topology tests for FRR built on top of Mininet.
11 Only tested with Ubuntu 16.04 and Ubuntu 18.04 (which uses Mininet 2.2.x).
13 Instructions are the same for all setups (i.e. ExaBGP is only used for BGP
16 Installing Mininet Infrastructure
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 apt-get install mininet
22 apt-get install python-pip
23 apt-get install iproute
26 pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet
28 useradd -d /var/run/exabgp/ -s /bin/false exabgp
33 Optional, will give better output.
38 disable apport (which move core files)
40 Set ``enabled=0`` in ``/etc/default/apport``.
42 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
44 #<domain> <type> <item> <value>
46 root soft core unlimited
48 root hard core unlimited
50 Reboot for options to take effect.
55 FRR needs to be installed separately. It is assume to be configured like the
56 standard Ubuntu Packages:
58 - Binaries in :file:`/usr/lib/frr`
59 - State Directory :file:`/var/run/frr`
60 - Running under user ``frr``, group ``frr``
61 - vtygroup: ``frrvty``
62 - config directory: :file:`/etc/frr`
63 - For FRR Packages, install the dbg package as well for coredump decoding
65 No FRR config needs to be done and no FRR daemons should be run ahead of the
66 test. They are all started as part of the test.
71 If you prefer to manually build FRR, then use the following suggested config:
77 --localstatedir=/var/run/frr \
78 --sbindir=/usr/lib/frr \
79 --sysconfdir=/etc/frr \
82 --enable-multipath=64 \
85 --enable-vty-group=frrvty \
86 --with-pkg-extra-version=-my-manual-build
88 And create ``frr`` user and ``frrvty`` group as follows:
92 addgroup --system --gid 92 frr
93 addgroup --system --gid 85 frrvty
94 adduser --system --ingroup frr --home /var/run/frr/ \
95 --gecos "FRRouting suite" --shell /bin/false frr
101 Execute all tests with output to console
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
106 py.test -s -v --tb=no
108 The above command must be executed from inside the topotests directory.
110 All test\_\* scripts in subdirectories are detected and executed (unless
111 disabled in ``pytest.ini`` file).
113 ``--tb=no`` disables the python traceback which might be irrelevant unless the
114 test script itself is debugged.
124 For example, and assuming you are inside the frr directory:
128 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
129 ./test_bgp_l3vpn_to_bgp_vrf.py
131 For further options, refer to pytest documentation.
133 Test will set exit code which can be used with ``git bisect``.
135 For the simulated topology, see the description in the python file.
137 If you need to clear the mininet setup between tests (if it isn't cleanly
138 shutdown), then use the ``mn -c`` command to clean up the environment.
140 StdErr log from daemos after exit
141 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
143 To enable the reporting of any messages seen on StdErr after the daemons exit,
144 the following env variable can be set::
146 export TOPOTESTS_CHECK_STDERR=Yes
148 (The value doesn't matter at this time. The check is whether the env
149 variable exists or not.) There is no pass/fail on this reporting; the
150 Output will be reported to the console.
152 Collect Memory Leak Information
153 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
155 FRR processes can report unfreed memory allocations upon exit. To
156 enable the reporting of memory leaks, define an environment variable
157 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
159 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
161 This will enable the check and output to console and the writing of
162 the information to files with the given prefix (followed by testname),
163 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
166 Running Topotests with AddressSanitizer
167 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
169 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
170 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
171 information on AddressSanitizer, see
172 https://github.com/google/sanitizers/wiki/AddressSanitizer.
174 The checks are done automatically in the library call of ``checkRouterRunning``
175 (ie at beginning of tests when there is a check for all daemons running). No
176 changes or extra configuration for topotests is required beside compiling the
177 suite with AddressSanitizer enabled.
179 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
180 If found, then this is added with context (calling test) to
181 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
183 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
184 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
185 for ``master`` branch:
189 git clone https://github.com/FRRouting/frr.git
193 export CFLAGS="-O1 -g -fsanitize=address -fno-omit-frame-pointer"
195 export LDFLAGS="-g -fsanitize=address -ldl"
196 ./configure --enable-shared=no \
197 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
198 --localstatedir=/var/run/frr \
199 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
200 --enable-exampledir=/usr/lib/frr/examples \
201 --with-moduledir=/usr/lib/frr/modules \
202 --enable-multipath=0 --enable-rtadv \
203 --enable-tcp-zebra --enable-fpm --enable-pimd \
207 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
208 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
210 and create ``frr`` user and ``frrvty`` group as shown above.
212 .. _topotests_docker:
214 Running Tests with Docker
215 -------------------------
217 There is a Docker image which allows to run topotests.
222 If you have Docker installed, you can run the topotests in Docker. The easiest
223 way to do this, is to use the make targets from this repository.
225 Your current user needs to have access to the Docker daemon. Alternatively you
226 can run these commands as root.
232 This command will pull the most recent topotests image from Dockerhub, compile
233 FRR inside of it, and run the topotests.
238 Internally, the topotests make target uses a shell script to pull the image and
239 spawn the Docker container.
241 There are several environment variables which can be used to modify the
242 behavior of the script, these can be listed by calling it with ``-h``:
246 ./tests/topotests/docker/frr-topotests.sh -h
248 For example, a volume is used to cache build artifacts between multiple runs of
249 the image. If you need to force a complete recompile, you can set
254 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
256 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
257 arguments and the first one starts with ``/`` or ``./``, they will replace the
258 call to pytest. If the appended arguments do not match this patttern, they will
259 be provided to pytest as arguments. So, to run a specific test with more
264 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
266 And to compile FRR but drop into a shell instead of running pytest:
270 ./tests/topotests/docker/frr-topotests.sh /bin/bash
275 The Docker image just includes all the components to run the topotests, but not
276 the topotests themselves. So if you just want to write tests and don't want to
277 make changes to the environment provided by the Docker image. You don't need to
278 build your own Docker image if you do not want to.
280 When developing new tests, there is one caveat though: The startup script of
281 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
282 pollution of the container with build artefacts from the host. This will also
283 result in your newly written tests being unavailable in the container unless at
284 least added to the index with ``git-add``.
286 If you do want to test changes to the Docker image, you can locally build the
287 image and run the tests without pulling from the registry using the following
293 TOPOTEST_PULL=0 make topotests
296 .. _topotests-guidelines:
304 To run the whole suite of tests the following commands must be executed at the
305 top level directory of topotest:
309 $ # Change to the top level directory of topotests.
310 $ cd path/to/topotests
311 $ # Tests must be run as root, since Mininet requires it.
314 In order to run a specific test, you can use the following command:
318 $ # running a specific topology
319 $ sudo pytest ospf-topo1/
320 $ # or inside the test folder
322 $ sudo pytest # to run all tests inside the directory
323 $ sudo pytest test_ospf_topo1.py # to run a specific test
324 $ # or outside the test folder
326 $ sudo pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
328 The output of the tested daemons will be available at the temporary folder of
333 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
335 zebra.err # zebra stderr output
336 zebra.log # zebra log file
337 zebra.out # zebra stdout output
340 You can also run memory leak tests to get reports:
344 $ # Set the environment variable to apply to a specific test...
345 $ sudo env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
346 $ # ...or apply to all tests adding this line to the configuration file
347 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
348 $ # You can also use your editor
350 $ # After running tests you should see your files:
351 $ ls /tmp/memleak_report_*
352 memleak_report_test_ospf_topo1.txt
357 This section will guide you in all recommended steps to produce a standard
360 This is the recommended test writing routine:
362 - Write a topology (Graphviz recommended)
363 - Obtain configuration files
364 - Write the test itself
365 - Create a Pull Request
367 Topotest File Hierarchy
368 """""""""""""""""""""""
370 Before starting to write any tests one must know the file hierarchy. The
371 repository hierarchy looks like this:
375 $ cd path/to/topotest
378 ./README.md # repository read me
379 ./GUIDELINES.md # this file
380 ./conftest.py # test hooks - pytest related functions
381 ./example-test # example test folder
382 ./example-test/__init__.py # python package marker - must always exist.
383 ./example-test/test_template.jpg # generated topology picture - see next section
384 ./example-test/test_template.dot # Graphviz dot file
385 ./example-test/test_template.py # the topology plus the test
387 ./ospf-topo1 # the ospf topology test
388 ./ospf-topo1/r1 # router 1 configuration files
389 ./ospf-topo1/r1/zebra.conf # zebra configuration file
390 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
391 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
392 # removed other for shortness sake
394 ./lib # shared test/topology functions
395 ./lib/topogen.py # topogen implementation
396 ./lib/topotest.py # topotest implementation
398 Guidelines for creating/editing topotest:
400 - New topologies that don't fit the existing directories should create its own
401 - Always remember to add the ``__init__.py`` to new folders, this makes auto
402 complete engines and pylint happy
403 - Router (Quagga/FRR) specific code should go on topotest.py
404 - Generic/repeated router actions should have an abstraction in
406 - Generic/repeated non-router code should go to topotest.py
407 - pytest related code should go to conftest.py (e.g. specialized asserts)
409 Defining the Topology
410 """""""""""""""""""""
412 The first step to write a new test is to define the topology. This step can be
413 done in many ways, but the recommended is to use Graphviz to generate a drawing
414 of the topology. It allows us to see the topology graphically and to see the
415 names of equipment, links and addresses.
417 Here is an example of Graphviz dot file that generates the template topology
418 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
419 get outdated, please see the linked file)::
441 label="s1\n192.168.0.0/24",
447 label="s2\n192.168.1.0/24",
453 r1 -- s1 [label="eth0\n.1"];
455 r1 -- s2 [label="eth1\n.100"];
456 r2 -- s2 [label="eth0\n.1"];
459 Here is the produced graph:
483 label="s1\n192.168.0.0/24",
489 label="s2\n192.168.1.0/24",
495 r1 -- s1 [label="eth0\n.1"];
497 r1 -- s2 [label="eth1\n.100"];
498 r2 -- s2 [label="eth0\n.1"];
501 Generating / Obtaining Configuration Files
502 """"""""""""""""""""""""""""""""""""""""""
504 In order to get the configuration files or command output for each router, we
505 need to run the topology and execute commands in ``vtysh``. The quickest way to
506 achieve that is writing the topology building code and running the topology.
508 To bootstrap your test topology, do the following steps:
510 - Copy the template test
515 $ touch new-topo/__init__.py
516 $ cp example-test/test_template.py new-topo/test_new_topo.py
518 - Modify the template according to your dot file
520 Here is the template topology described in the previous section in python code:
524 class TemplateTopo(Topo):
525 "Test topology builder"
526 def build(self, *_args, **_opts):
528 tgen = get_topogen(self)
531 for routern in range(1, 3):
532 tgen.add_router('r{}'.format(routern))
534 # Create a switch with just one router connected to it to simulate a
536 switch = tgen.add_switch('s1')
537 switch.add_link(tgen.gears['r1'])
539 # Create a connection between r1 and r2
540 switch = tgen.add_switch('s2')
541 switch.add_link(tgen.gears['r1'])
542 switch.add_link(tgen.gears['r2'])
546 Topogen allows us to run the topology without running any tests, you can do
547 that using the following example commands:
551 $ # Running your bootstraped topology
552 $ sudo pytest -s --topology-only new-topo/test_new_topo.py
553 $ # Running the test_template.py topology
554 $ sudo pytest -s --topology-only example-test/test_template.py
555 $ # Running the ospf_topo1.py topology
556 $ sudo pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
558 Parameters explanation:
564 Actives input/output capture. This is required by mininet in order to show
565 the interactive shell.
567 .. option:: --topology-only
569 Don't run any tests, just build the topology.
571 After executing the commands above, you should get the following terminal
576 === test session starts ===
577 platform linux2 -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
578 rootdir: /media/sf_src/topotests, inifile: pytest.ini
581 ospf-topo1/test_ospf_topo1.py *** Starting controller
583 *** Starting 6 switches
584 switch1 switch2 switch3 switch4 switch5 switch6 ...
585 r2: frr zebra started
586 r2: frr ospfd started
587 r3: frr zebra started
588 r3: frr ospfd started
589 r1: frr zebra started
590 r1: frr ospfd started
591 r4: frr zebra started
592 r4: frr ospfd started
596 The last line shows us that we are now using the Mininet CLI (Command Line
597 Interface), from here you can call your router ``vtysh`` or even bash.
599 Here are some commands example:
603 mininet> r1 ping 10.0.3.1
604 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
605 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
606 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
607 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
609 --- 10.0.3.1 ping statistics ---
610 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
611 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
615 mininet> r1 ping 10.0.3.3
616 PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
617 64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.87 ms
618 64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.080 ms
619 64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=0.091 ms
621 --- 10.0.3.3 ping statistics ---
622 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
623 rtt min/avg/max/mdev = 0.080/1.014/2.872/1.313 ms
629 Hello, this is FRRouting (version 3.1-devrzalamena-build).
630 Copyright 1996-2005 Kunihiro Ishiguro, et al.
632 frr-1# show running-config
633 Building configuration...
635 Current configuration:
637 frr version 3.1-devrzalamena-build
638 frr defaults traditional
640 no service integrated-vtysh-config
647 ip address 10.0.3.1/24
650 ip address 10.0.10.1/24
653 ip address 172.16.0.2/24
656 ospf router-id 10.0.255.3
658 redistribute connected
660 network 10.0.3.0/24 area 0
661 network 10.0.10.0/24 area 0
662 network 172.16.0.0/24 area 1
669 After you successfully configured your topology, you can obtain the
670 configuration files (per-daemon) using the following commands:
674 mininet> r3 vtysh -d ospfd
676 Hello, this is FRRouting (version 3.1-devrzalamena-build).
677 Copyright 1996-2005 Kunihiro Ishiguro, et al.
679 frr-1# show running-config
680 Building configuration...
682 Current configuration:
684 frr version 3.1-devrzalamena-build
685 frr defaults traditional
686 no service integrated-vtysh-config
691 ospf router-id 10.0.255.3
693 redistribute connected
695 network 10.0.3.0/24 area 0
696 network 10.0.10.0/24 area 0
697 network 172.16.0.0/24 area 1
707 Test topologies should always be bootstrapped from
708 :file:`tests/topotests/example-test/test_template.py` because it contains
709 important boilerplate code that can't be avoided, like:
711 - imports: os, sys, pytest, topotest/topogen and mininet topology class
712 - The global variable CWD (Current Working directory): which is most likely
713 going to be used to reference the routers configuration file location
719 # For all registered routers, load the zebra configuration file
720 for rname, router in router_list.iteritems():
723 os.path.join(CWD, '{}/zebra.conf'.format(rname))
725 # os.path.join() joins the CWD string with arguments adding the necessary
726 # slashes ('/'). Arguments must not begin with '/'.
728 - The topology class that inherits from Mininet Topo class:
732 class TemplateTopo(Topo):
733 def build(self, *_args, **_opts):
734 tgen = get_topogen(self)
735 # topology build code
737 - pytest ``setup_module()`` and ``teardown_module()`` to start the topology
741 def setup_module(_m):
742 tgen = Topogen(TemplateTopo)
743 tgen.start_topology('debug')
745 def teardown_module(_m):
749 - ``__main__`` initialization code (to support running the script directly)
753 if __name__ == '__main__':
754 sys.exit(pytest.main(["-s"]))
758 - Test code should always be declared inside functions that begin with the
759 ``test_`` prefix. Functions beginning with different prefixes will not be run
761 - Configuration files and long output commands should go into separated files
762 inside folders named after the equipment.
763 - Tests must be able to run without any interaction. To make sure your test
764 conforms with this, run it without the :option:`-s` parameter.
768 - Keep results in stack variables, so people inspecting code with ``pdb`` can
769 easily print their values.
775 assert foobar(router1, router2)
781 result = foobar(router1, router2)
784 - Use ``assert`` messages to indicate where the test failed.
790 for router in router_list:
792 assert condition, 'Router "{}" condition failed'.format(router.name)
797 The most effective ways to inspect topology tests are:
799 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
800 appear when an assertion fails
802 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
804 - Set a breakpoint in the test code with ``pdb``
810 # Add the pdb import at the beginning of the file
814 # Add a breakpoint where you think the problem is
820 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
821 shell allows us to run many useful operations like:
823 - Setting breaking point on file/function/conditions (e.g. ``break``,
825 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
826 - Running python code
830 The TopoGear (equipment abstraction class) implements the ``__str__`` method
831 that allows the user to inspect equipment information.
833 Example of pdb usage:
837 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
838 -> for rnum in range(1, 5):
840 Documented commands (type help <topic>):
841 ========================================
842 EOF bt cont enable jump pp run unt
843 a c continue exit l q s until
844 alias cl d h list quit step up
845 args clear debug help n r tbreak w
846 b commands disable ignore next restart u whatis
847 break condition down j p return unalias where
849 Miscellaneous help topics:
850 ==========================
853 Undocumented commands:
854 ======================
858 116 title2="Expected output")
860 118 def test_ospf_convergence():
861 119 "Test OSPF daemon convergence"
863 121 -> for rnum in range(1, 5):
864 122 router = 'r{}'.format(rnum)
866 124 # Load expected results from the command
867 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
868 126 expected = open(reffile).read()
870 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
871 -> router = 'r{}'.format(rnum)
873 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
874 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
879 (Pdb) tgen = get_topogen()
880 (Pdb) pp tgen.gears[router]
881 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
882 (Pdb) pp str(tgen.gears[router])
883 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
886 121 for rnum in range(1, 5):
887 122 router = 'r{}'.format(rnum)
889 124 # Load expected results from the command
890 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
891 126 expected = open(reffile).read()
893 128 # Run test function until we get an result. Wait at most 60 seconds.
894 129 test_func = partial(compare_show_ip_ospf, router, expected)
895 130 result, diff = topotest.run_and_expect(test_func, '',
896 (Pdb) router1 = tgen.gears[router]
897 (Pdb) router1.vtysh_cmd('show ip ospf route')
898 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
899 (Pdb) tgen.mininet_cli()
903 To enable more debug messages in other Topogen subsystems (like Mininet), more
904 logging messages can be displayed by modifying the test configuration file
910 # Change the default verbosity line from 'info'...
915 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
916 To learn/remember common code snippets see :ref:`topotests-snippets`.
918 Before creating a new topology, make sure that there isn't one already that
919 does what you need. If nothing is similar, then you may create a new topology,
920 preferably, using the newest template
921 (:file:`tests/topotests/example-test/test_template.py`).
923 .. include:: topotests-snippets.rst
928 All the configs and scripts are licensed under a ISC-style license. See Python