6 Topotests is a suite of topology tests for FRR built on top of Mininet.
11 Only tested with Ubuntu 16.04 and Ubuntu 18.04 (which uses Mininet 2.2.x).
13 Instructions are the same for all setups (i.e. ExaBGP is only used for BGP
16 Installing Mininet Infrastructure
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 apt-get install mininet
22 apt-get install python-pip
23 apt-get install iproute
26 pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet
28 useradd -d /var/run/exabgp/ -s /bin/false exabgp
33 Optional, will give better output.
38 disable apport (which move core files)
40 Set ``enabled=0`` in ``/etc/default/apport``.
42 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
44 #<domain> <type> <item> <value>
46 root soft core unlimited
48 root hard core unlimited
50 Reboot for options to take effect.
55 FRR needs to be installed separately. It is assume to be configured like the
56 standard Ubuntu Packages:
58 - Binaries in :file:`/usr/lib/frr`
59 - State Directory :file:`/var/run/frr`
60 - Running under user ``frr``, group ``frr``
61 - vtygroup: ``frrvty``
62 - config directory: :file:`/etc/frr`
63 - For FRR Packages, install the dbg package as well for coredump decoding
65 No FRR config needs to be done and no FRR daemons should be run ahead of the
66 test. They are all started as part of the test.
71 If you prefer to manually build FRR, then use the following suggested config:
77 --localstatedir=/var/run/frr \
78 --sbindir=/usr/lib/frr \
79 --sysconfdir=/etc/frr \
82 --enable-multipath=64 \
85 --enable-vty-group=frrvty \
86 --with-pkg-extra-version=-my-manual-build
88 And create ``frr`` user and ``frrvty`` group as follows:
92 addgroup --system --gid 92 frr
93 addgroup --system --gid 85 frrvty
94 adduser --system --ingroup frr --home /var/run/frr/ \
95 --gecos "FRRouting suite" --shell /bin/false frr
101 Execute all tests with output to console
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
106 py.test -s -v --tb=no
108 The above command must be executed from inside the topotests directory.
110 All test\_\* scripts in subdirectories are detected and executed (unless
111 disabled in ``pytest.ini`` file).
113 ``--tb=no`` disables the python traceback which might be irrelevant unless the
114 test script itself is debugged.
124 For example, and assuming you are inside the frr directory:
128 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
129 ./test_bgp_l3vpn_to_bgp_vrf.py
131 For further options, refer to pytest documentation.
133 Test will set exit code which can be used with ``git bisect``.
135 For the simulated topology, see the description in the python file.
137 If you need to clear the mininet setup between tests (if it isn't cleanly
138 shutdown), then use the ``mn -c`` command to clean up the environment.
140 StdErr log from daemos after exit
141 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
143 To enable the reporting of any messages seen on StdErr after the daemons exit,
144 the following env variable can be set::
146 export TOPOTESTS_CHECK_STDERR=Yes
148 (The value doesn't matter at this time. The check is if the env variable exists
149 or not) There is no pass/fail on this reporting. The Output will be reported to
152 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
154 This will enable the check and output to console and the writing of the
155 information to files with the given prefix (followed by testname), ie
156 :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case of a memory
159 Collect Memory Leak Information
160 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
162 FRR processes have the capabilities to report remaining memory allocations upon
163 exit. To enable the reporting of the memory, define an environment variable
164 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
166 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
168 This will enable the check and output to console and the writing of the
169 information to files with the given prefix (followed by testname), ie
170 :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case of a memory
173 Running Topotests with AddressSanitizer
174 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
176 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
177 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
178 information on AddressSanitizer, see
179 https://github.com/google/sanitizers/wiki/AddressSanitizer.
181 The checks are done automatically in the library call of ``checkRouterRunning``
182 (ie at beginning of tests when there is a check for all daemons running). No
183 changes or extra configuration for topotests is required beside compiling the
184 suite with AddressSanitizer enabled.
186 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
187 If found, then this is added with context (calling test) to
188 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
190 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
191 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
192 for ``master`` branch:
196 git clone https://github.com/FRRouting/frr.git
200 export CFLAGS="-O1 -g -fsanitize=address -fno-omit-frame-pointer"
202 export LDFLAGS="-g -fsanitize=address -ldl"
203 ./configure --enable-shared=no \
204 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
205 --localstatedir=/var/run/frr \
206 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
207 --enable-exampledir=/usr/lib/frr/examples \
208 --with-moduledir=/usr/lib/frr/modules \
209 --enable-multipath=0 --enable-rtadv \
210 --enable-tcp-zebra --enable-fpm --enable-pimd \
214 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
215 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
217 and create ``frr`` user and ``frrvty`` group as shown above.
219 .. _topotests_docker:
221 Running Tests with Docker
222 -------------------------
224 There is a Docker image which allows to run topotests.
229 If you have Docker installed, you can run the topotests in Docker. The easiest
230 way to do this, is to use the make targets from this repository.
232 Your current user needs to have access to the Docker daemon. Alternatively you
233 can run these commands as root.
239 This command will pull the most recent topotests image from Dockerhub, compile
240 FRR inside of it, and run the topotests.
245 Internally, the topotests make target uses a shell script to pull the image and
246 spawn the Docker container.
248 There are several environment variables which can be used to modify the
249 behavior of the script, these can be listed by calling it with ``-h``:
253 ./tests/topotests/docker/frr-topotests.sh -h
255 For example, a volume is used to cache build artifacts between multiple runs of
256 the image. If you need to force a complete recompile, you can set
261 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
263 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
264 arguments and the first one starts with ``/`` or ``./``, they will replace the
265 call to pytest. If the appended arguments do not match this patttern, they will
266 be provided to pytest as arguments. So, to run a specific test with more
271 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
273 And to compile FRR but drop into a shell instead of running pytest:
277 ./tests/topotests/docker/frr-topotests.sh /bin/bash
282 The Docker image just includes all the components to run the topotests, but not
283 the topotests themselves. So if you just want to write tests and don't want to
284 make changes to the environment provided by the Docker image. You don't need to
285 build your own Docker image if you do not want to.
287 When developing new tests, there is one caveat though: The startup script of
288 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
289 pollution of the container with build artefacts from the host. This will also
290 result in your newly written tests being unavailable in the container unless at
291 least added to the index with ``git-add``.
293 If you do want to test changes to the Docker image, you can locally build the
294 image and run the tests without pulling from the registry using the following
300 TOPOTEST_PULL=0 make topotests
303 .. _topotests-guidelines:
311 To run the whole suite of tests the following commands must be executed at the
312 top level directory of topotest:
316 $ # Change to the top level directory of topotests.
317 $ cd path/to/topotests
318 $ # Tests must be run as root, since Mininet requires it.
321 In order to run a specific test, you can use the following command:
325 $ # running a specific topology
326 $ sudo pytest ospf-topo1/
327 $ # or inside the test folder
329 $ sudo pytest # to run all tests inside the directory
330 $ sudo pytest test_ospf_topo1.py # to run a specific test
331 $ # or outside the test folder
333 $ sudo pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
335 The output of the tested daemons will be available at the temporary folder of
340 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
342 zebra.err # zebra stderr output
343 zebra.log # zebra log file
344 zebra.out # zebra stdout output
347 You can also run memory leak tests to get reports:
351 $ # Set the environment variable to apply to a specific test...
352 $ sudo env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
353 $ # ...or apply to all tests adding this line to the configuration file
354 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
355 $ # You can also use your editor
357 $ # After running tests you should see your files:
358 $ ls /tmp/memleak_report_*
359 memleak_report_test_ospf_topo1.txt
364 This section will guide you in all recommended steps to produce a standard
367 This is the recommended test writing routine:
369 - Write a topology (Graphviz recommended)
370 - Obtain configuration files
371 - Write the test itself
372 - Create a Pull Request
374 Topotest File Hierarchy
375 """""""""""""""""""""""
377 Before starting to write any tests one must know the file hierarchy. The
378 repository hierarchy looks like this:
382 $ cd path/to/topotest
385 ./README.md # repository read me
386 ./GUIDELINES.md # this file
387 ./conftest.py # test hooks - pytest related functions
388 ./example-test # example test folder
389 ./example-test/__init__.py # python package marker - must always exist.
390 ./example-test/test_template.jpg # generated topology picture - see next section
391 ./example-test/test_template.dot # Graphviz dot file
392 ./example-test/test_template.py # the topology plus the test
394 ./ospf-topo1 # the ospf topology test
395 ./ospf-topo1/r1 # router 1 configuration files
396 ./ospf-topo1/r1/zebra.conf # zebra configuration file
397 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
398 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
399 # removed other for shortness sake
401 ./lib # shared test/topology functions
402 ./lib/topogen.py # topogen implementation
403 ./lib/topotest.py # topotest implementation
405 Guidelines for creating/editing topotest:
407 - New topologies that don't fit the existing directories should create its own
408 - Always remember to add the ``__init__.py`` to new folders, this makes auto
409 complete engines and pylint happy
410 - Router (Quagga/FRR) specific code should go on topotest.py
411 - Generic/repeated router actions should have an abstraction in
413 - Generic/repeated non-router code should go to topotest.py
414 - pytest related code should go to conftest.py (e.g. specialized asserts)
416 Defining the Topology
417 """""""""""""""""""""
419 The first step to write a new test is to define the topology. This step can be
420 done in many ways, but the recommended is to use Graphviz to generate a drawing
421 of the topology. It allows us to see the topology graphically and to see the
422 names of equipment, links and addresses.
424 Here is an example of Graphviz dot file that generates the template topology
425 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
426 get outdated, please see the linked file)::
448 label="s1\n192.168.0.0/24",
454 label="s2\n192.168.1.0/24",
460 r1 -- s1 [label="eth0\n.1"];
462 r1 -- s2 [label="eth1\n.100"];
463 r2 -- s2 [label="eth0\n.1"];
466 Here is the produced graph:
490 label="s1\n192.168.0.0/24",
496 label="s2\n192.168.1.0/24",
502 r1 -- s1 [label="eth0\n.1"];
504 r1 -- s2 [label="eth1\n.100"];
505 r2 -- s2 [label="eth0\n.1"];
508 Generating / Obtaining Configuration Files
509 """"""""""""""""""""""""""""""""""""""""""
511 In order to get the configuration files or command output for each router, we
512 need to run the topology and execute commands in ``vtysh``. The quickest way to
513 achieve that is writing the topology building code and running the topology.
515 To bootstrap your test topology, do the following steps:
517 - Copy the template test
522 $ touch new-topo/__init__.py
523 $ cp example-test/test_template.py new-topo/test_new_topo.py
525 - Modify the template according to your dot file
527 Here is the template topology described in the previous section in python code:
531 class TemplateTopo(Topo):
532 "Test topology builder"
533 def build(self, *_args, **_opts):
535 tgen = get_topogen(self)
538 for routern in range(1, 3):
539 tgen.add_router('r{}'.format(routern))
541 # Create a switch with just one router connected to it to simulate a
543 switch = tgen.add_switch('s1')
544 switch.add_link(tgen.gears['r1'])
546 # Create a connection between r1 and r2
547 switch = tgen.add_switch('s2')
548 switch.add_link(tgen.gears['r1'])
549 switch.add_link(tgen.gears['r2'])
553 Topogen allows us to run the topology without running any tests, you can do
554 that using the following example commands:
558 $ # Running your bootstraped topology
559 $ sudo pytest -s --topology-only new-topo/test_new_topo.py
560 $ # Running the test_template.py topology
561 $ sudo pytest -s --topology-only example-test/test_template.py
562 $ # Running the ospf_topo1.py topology
563 $ sudo pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
565 Parameters explanation:
571 Actives input/output capture. This is required by mininet in order to show
572 the interactive shell.
574 .. option:: --topology-only
576 Don't run any tests, just build the topology.
578 After executing the commands above, you should get the following terminal
583 === test session starts ===
584 platform linux2 -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
585 rootdir: /media/sf_src/topotests, inifile: pytest.ini
588 ospf-topo1/test_ospf_topo1.py *** Starting controller
590 *** Starting 6 switches
591 switch1 switch2 switch3 switch4 switch5 switch6 ...
592 r2: frr zebra started
593 r2: frr ospfd started
594 r3: frr zebra started
595 r3: frr ospfd started
596 r1: frr zebra started
597 r1: frr ospfd started
598 r4: frr zebra started
599 r4: frr ospfd started
603 The last line shows us that we are now using the Mininet CLI (Command Line
604 Interface), from here you can call your router ``vtysh`` or even bash.
606 Here are some commands example:
610 mininet> r1 ping 10.0.3.1
611 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
612 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
613 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
614 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
616 --- 10.0.3.1 ping statistics ---
617 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
618 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
622 mininet> r1 ping 10.0.3.3
623 PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
624 64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.87 ms
625 64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.080 ms
626 64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=0.091 ms
628 --- 10.0.3.3 ping statistics ---
629 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
630 rtt min/avg/max/mdev = 0.080/1.014/2.872/1.313 ms
636 Hello, this is FRRouting (version 3.1-devrzalamena-build).
637 Copyright 1996-2005 Kunihiro Ishiguro, et al.
639 frr-1# show running-config
640 Building configuration...
642 Current configuration:
644 frr version 3.1-devrzalamena-build
645 frr defaults traditional
647 no service integrated-vtysh-config
654 ip address 10.0.3.1/24
657 ip address 10.0.10.1/24
660 ip address 172.16.0.2/24
663 ospf router-id 10.0.255.3
665 redistribute connected
667 network 10.0.3.0/24 area 0
668 network 10.0.10.0/24 area 0
669 network 172.16.0.0/24 area 1
676 After you successfully configured your topology, you can obtain the
677 configuration files (per-daemon) using the following commands:
681 mininet> r3 vtysh -d ospfd
683 Hello, this is FRRouting (version 3.1-devrzalamena-build).
684 Copyright 1996-2005 Kunihiro Ishiguro, et al.
686 frr-1# show running-config
687 Building configuration...
689 Current configuration:
691 frr version 3.1-devrzalamena-build
692 frr defaults traditional
693 no service integrated-vtysh-config
698 ospf router-id 10.0.255.3
700 redistribute connected
702 network 10.0.3.0/24 area 0
703 network 10.0.10.0/24 area 0
704 network 172.16.0.0/24 area 1
714 Test topologies should always be bootstrapped from
715 :file:`tests/topotests/example-test/test_template.py` because it contains
716 important boilerplate code that can't be avoided, like:
718 - imports: os, sys, pytest, topotest/topogen and mininet topology class
719 - The global variable CWD (Current Working directory): which is most likely
720 going to be used to reference the routers configuration file location
726 # For all registered routers, load the zebra configuration file
727 for rname, router in router_list.iteritems():
730 os.path.join(CWD, '{}/zebra.conf'.format(rname))
732 # os.path.join() joins the CWD string with arguments adding the necessary
733 # slashes ('/'). Arguments must not begin with '/'.
735 - The topology class that inherits from Mininet Topo class:
739 class TemplateTopo(Topo):
740 def build(self, *_args, **_opts):
741 tgen = get_topogen(self)
742 # topology build code
744 - pytest ``setup_module()`` and ``teardown_module()`` to start the topology
748 def setup_module(_m):
749 tgen = Topogen(TemplateTopo)
750 tgen.start_topology('debug')
752 def teardown_module(_m):
756 - ``__main__`` initialization code (to support running the script directly)
760 if __name__ == '__main__':
761 sys.exit(pytest.main(["-s"]))
765 - Test code should always be declared inside functions that begin with the
766 ``test_`` prefix. Functions beginning with different prefixes will not be run
768 - Configuration files and long output commands should go into separated files
769 inside folders named after the equipment.
770 - Tests must be able to run without any interaction. To make sure your test
771 conforms with this, run it without the :option:`-s` parameter.
775 - Keep results in stack variables, so people inspecting code with ``pdb`` can
776 easily print their values.
782 assert foobar(router1, router2)
788 result = foobar(router1, router2)
791 - Use ``assert`` messages to indicate where the test failed.
797 for router in router_list:
799 assert condition, 'Router "{}" condition failed'.format(router.name)
804 The most effective ways to inspect topology tests are:
806 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
807 appear when an assertion fails
809 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
811 - Set a breakpoint in the test code with ``pdb``
817 # Add the pdb import at the beginning of the file
821 # Add a breakpoint where you think the problem is
827 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
828 shell allows us to run many useful operations like:
830 - Setting breaking point on file/function/conditions (e.g. ``break``,
832 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
833 - Running python code
837 The TopoGear (equipment abstraction class) implements the ``__str__`` method
838 that allows the user to inspect equipment information.
840 Example of pdb usage:
844 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
845 -> for rnum in range(1, 5):
847 Documented commands (type help <topic>):
848 ========================================
849 EOF bt cont enable jump pp run unt
850 a c continue exit l q s until
851 alias cl d h list quit step up
852 args clear debug help n r tbreak w
853 b commands disable ignore next restart u whatis
854 break condition down j p return unalias where
856 Miscellaneous help topics:
857 ==========================
860 Undocumented commands:
861 ======================
865 116 title2="Expected output")
867 118 def test_ospf_convergence():
868 119 "Test OSPF daemon convergence"
870 121 -> for rnum in range(1, 5):
871 122 router = 'r{}'.format(rnum)
873 124 # Load expected results from the command
874 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
875 126 expected = open(reffile).read()
877 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
878 -> router = 'r{}'.format(rnum)
880 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
881 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
886 (Pdb) tgen = get_topogen()
887 (Pdb) pp tgen.gears[router]
888 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
889 (Pdb) pp str(tgen.gears[router])
890 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
893 121 for rnum in range(1, 5):
894 122 router = 'r{}'.format(rnum)
896 124 # Load expected results from the command
897 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
898 126 expected = open(reffile).read()
900 128 # Run test function until we get an result. Wait at most 60 seconds.
901 129 test_func = partial(compare_show_ip_ospf, router, expected)
902 130 result, diff = topotest.run_and_expect(test_func, '',
903 (Pdb) router1 = tgen.gears[router]
904 (Pdb) router1.vtysh_cmd('show ip ospf route')
905 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
906 (Pdb) tgen.mininet_cli()
910 To enable more debug messages in other Topogen subsystems (like Mininet), more
911 logging messages can be displayed by modifying the test configuration file
917 # Change the default verbosity line from 'info'...
922 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
923 To learn/remember common code snippets see :ref:`topotests-snippets`.
925 Before creating a new topology, make sure that there isn't one already that
926 does what you need. If nothing is similar, then you may create a new topology,
927 preferably, using the newest template
928 (:file:`tests/topotests/example-test/test_template.py`).
930 .. include:: topotests-snippets.rst
935 All the configs and scripts are licensed under a ISC-style license. See Python