6 Topotests is a suite of topology tests for FRR built on top of Mininet.
11 Only tested with Ubuntu 16.04 and Ubuntu 18.04 (which uses Mininet 2.2.x).
13 Instructions are the same for all setups (i.e. ExaBGP is only used for BGP
16 Installing Mininet Infrastructure
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 apt-get install mininet
22 apt-get install python-pip
23 apt-get install iproute
25 pip install "pytest<5"
26 pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet
28 useradd -d /var/run/exabgp/ -s /bin/false exabgp
33 Optional, will give better output.
38 disable apport (which move core files)
40 Set ``enabled=0`` in ``/etc/default/apport``.
42 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
44 #<domain> <type> <item> <value>
46 root soft core unlimited
48 root hard core unlimited
50 Reboot for options to take effect.
55 FRR needs to be installed separately. It is assume to be configured like the
56 standard Ubuntu Packages:
58 - Binaries in :file:`/usr/lib/frr`
59 - State Directory :file:`/var/run/frr`
60 - Running under user ``frr``, group ``frr``
61 - vtygroup: ``frrvty``
62 - config directory: :file:`/etc/frr`
63 - For FRR Packages, install the dbg package as well for coredump decoding
65 No FRR config needs to be done and no FRR daemons should be run ahead of the
66 test. They are all started as part of the test.
71 If you prefer to manually build FRR, then use the following suggested config:
77 --localstatedir=/var/run/frr \
78 --sbindir=/usr/lib/frr \
79 --sysconfdir=/etc/frr \
82 --enable-multipath=64 \
85 --enable-vty-group=frrvty \
86 --with-pkg-extra-version=-my-manual-build
88 And create ``frr`` user and ``frrvty`` group as follows:
92 addgroup --system --gid 92 frr
93 addgroup --system --gid 85 frrvty
94 adduser --system --ingroup frr --home /var/run/frr/ \
95 --gecos "FRRouting suite" --shell /bin/false frr
101 Execute all tests with output to console
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
106 py.test -s -v --tb=no
108 The above command must be executed from inside the topotests directory.
110 All test\_\* scripts in subdirectories are detected and executed (unless
111 disabled in ``pytest.ini`` file).
113 ``--tb=no`` disables the python traceback which might be irrelevant unless the
114 test script itself is debugged.
124 For example, and assuming you are inside the frr directory:
128 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
129 ./test_bgp_l3vpn_to_bgp_vrf.py
131 For further options, refer to pytest documentation.
133 Test will set exit code which can be used with ``git bisect``.
135 For the simulated topology, see the description in the python file.
137 If you need to clear the mininet setup between tests (if it isn't cleanly
138 shutdown), then use the ``mn -c`` command to clean up the environment.
140 StdErr log from daemos after exit
141 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
143 To enable the reporting of any messages seen on StdErr after the daemons exit,
144 the following env variable can be set::
146 export TOPOTESTS_CHECK_STDERR=Yes
148 (The value doesn't matter at this time. The check is whether the env
149 variable exists or not.) There is no pass/fail on this reporting; the
150 Output will be reported to the console.
152 Collect Memory Leak Information
153 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
155 FRR processes can report unfreed memory allocations upon exit. To
156 enable the reporting of memory leaks, define an environment variable
157 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
159 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
161 This will enable the check and output to console and the writing of
162 the information to files with the given prefix (followed by testname),
163 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
166 Running Topotests with AddressSanitizer
167 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
169 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
170 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
171 information on AddressSanitizer, see
172 https://github.com/google/sanitizers/wiki/AddressSanitizer.
174 The checks are done automatically in the library call of ``checkRouterRunning``
175 (ie at beginning of tests when there is a check for all daemons running). No
176 changes or extra configuration for topotests is required beside compiling the
177 suite with AddressSanitizer enabled.
179 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
180 If found, then this is added with context (calling test) to
181 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
183 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
184 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
185 for ``master`` branch:
189 git clone https://github.com/FRRouting/frr.git
193 --enable-address-sanitizer \
194 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
195 --localstatedir=/var/run/frr \
196 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
197 --enable-exampledir=/usr/lib/frr/examples \
198 --with-moduledir=/usr/lib/frr/modules \
199 --enable-multipath=0 --enable-rtadv \
200 --enable-tcp-zebra --enable-fpm --enable-pimd \
204 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
205 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
207 and create ``frr`` user and ``frrvty`` group as shown above.
209 .. _topotests_docker:
211 Running Tests with Docker
212 -------------------------
214 There is a Docker image which allows to run topotests.
219 If you have Docker installed, you can run the topotests in Docker. The easiest
220 way to do this, is to use the make targets from this repository.
222 Your current user needs to have access to the Docker daemon. Alternatively you
223 can run these commands as root.
229 This command will pull the most recent topotests image from Dockerhub, compile
230 FRR inside of it, and run the topotests.
235 Internally, the topotests make target uses a shell script to pull the image and
236 spawn the Docker container.
238 There are several environment variables which can be used to modify the
239 behavior of the script, these can be listed by calling it with ``-h``:
243 ./tests/topotests/docker/frr-topotests.sh -h
245 For example, a volume is used to cache build artifacts between multiple runs of
246 the image. If you need to force a complete recompile, you can set
251 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
253 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
254 arguments and the first one starts with ``/`` or ``./``, they will replace the
255 call to pytest. If the appended arguments do not match this patttern, they will
256 be provided to pytest as arguments. So, to run a specific test with more
261 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
263 And to compile FRR but drop into a shell instead of running pytest:
267 ./tests/topotests/docker/frr-topotests.sh /bin/bash
272 The Docker image just includes all the components to run the topotests, but not
273 the topotests themselves. So if you just want to write tests and don't want to
274 make changes to the environment provided by the Docker image. You don't need to
275 build your own Docker image if you do not want to.
277 When developing new tests, there is one caveat though: The startup script of
278 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
279 pollution of the container with build artefacts from the host. This will also
280 result in your newly written tests being unavailable in the container unless at
281 least added to the index with ``git-add``.
283 If you do want to test changes to the Docker image, you can locally build the
284 image and run the tests without pulling from the registry using the following
290 TOPOTEST_PULL=0 make topotests
293 .. _topotests-guidelines:
301 To run the whole suite of tests the following commands must be executed at the
302 top level directory of topotest:
306 $ # Change to the top level directory of topotests.
307 $ cd path/to/topotests
308 $ # Tests must be run as root, since Mininet requires it.
311 In order to run a specific test, you can use the following command:
315 $ # running a specific topology
316 $ sudo pytest ospf-topo1/
317 $ # or inside the test folder
319 $ sudo pytest # to run all tests inside the directory
320 $ sudo pytest test_ospf_topo1.py # to run a specific test
321 $ # or outside the test folder
323 $ sudo pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
325 The output of the tested daemons will be available at the temporary folder of
330 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
332 zebra.err # zebra stderr output
333 zebra.log # zebra log file
334 zebra.out # zebra stdout output
337 You can also run memory leak tests to get reports:
341 $ # Set the environment variable to apply to a specific test...
342 $ sudo env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
343 $ # ...or apply to all tests adding this line to the configuration file
344 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
345 $ # You can also use your editor
347 $ # After running tests you should see your files:
348 $ ls /tmp/memleak_report_*
349 memleak_report_test_ospf_topo1.txt
354 This section will guide you in all recommended steps to produce a standard
357 This is the recommended test writing routine:
359 - Write a topology (Graphviz recommended)
360 - Obtain configuration files
361 - Write the test itself
362 - Create a Pull Request
364 Topotest File Hierarchy
365 """""""""""""""""""""""
367 Before starting to write any tests one must know the file hierarchy. The
368 repository hierarchy looks like this:
372 $ cd path/to/topotest
375 ./README.md # repository read me
376 ./GUIDELINES.md # this file
377 ./conftest.py # test hooks - pytest related functions
378 ./example-test # example test folder
379 ./example-test/__init__.py # python package marker - must always exist.
380 ./example-test/test_template.jpg # generated topology picture - see next section
381 ./example-test/test_template.dot # Graphviz dot file
382 ./example-test/test_template.py # the topology plus the test
384 ./ospf-topo1 # the ospf topology test
385 ./ospf-topo1/r1 # router 1 configuration files
386 ./ospf-topo1/r1/zebra.conf # zebra configuration file
387 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
388 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
389 # removed other for shortness sake
391 ./lib # shared test/topology functions
392 ./lib/topogen.py # topogen implementation
393 ./lib/topotest.py # topotest implementation
395 Guidelines for creating/editing topotest:
397 - New topologies that don't fit the existing directories should create its own
398 - Always remember to add the ``__init__.py`` to new folders, this makes auto
399 complete engines and pylint happy
400 - Router (Quagga/FRR) specific code should go on topotest.py
401 - Generic/repeated router actions should have an abstraction in
403 - Generic/repeated non-router code should go to topotest.py
404 - pytest related code should go to conftest.py (e.g. specialized asserts)
406 Defining the Topology
407 """""""""""""""""""""
409 The first step to write a new test is to define the topology. This step can be
410 done in many ways, but the recommended is to use Graphviz to generate a drawing
411 of the topology. It allows us to see the topology graphically and to see the
412 names of equipment, links and addresses.
414 Here is an example of Graphviz dot file that generates the template topology
415 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
416 get outdated, please see the linked file)::
438 label="s1\n192.168.0.0/24",
444 label="s2\n192.168.1.0/24",
450 r1 -- s1 [label="eth0\n.1"];
452 r1 -- s2 [label="eth1\n.100"];
453 r2 -- s2 [label="eth0\n.1"];
456 Here is the produced graph:
480 label="s1\n192.168.0.0/24",
486 label="s2\n192.168.1.0/24",
492 r1 -- s1 [label="eth0\n.1"];
494 r1 -- s2 [label="eth1\n.100"];
495 r2 -- s2 [label="eth0\n.1"];
498 Generating / Obtaining Configuration Files
499 """"""""""""""""""""""""""""""""""""""""""
501 In order to get the configuration files or command output for each router, we
502 need to run the topology and execute commands in ``vtysh``. The quickest way to
503 achieve that is writing the topology building code and running the topology.
505 To bootstrap your test topology, do the following steps:
507 - Copy the template test
512 $ touch new-topo/__init__.py
513 $ cp example-test/test_template.py new-topo/test_new_topo.py
515 - Modify the template according to your dot file
517 Here is the template topology described in the previous section in python code:
521 class TemplateTopo(Topo):
522 "Test topology builder"
523 def build(self, *_args, **_opts):
525 tgen = get_topogen(self)
528 for routern in range(1, 3):
529 tgen.add_router('r{}'.format(routern))
531 # Create a switch with just one router connected to it to simulate a
533 switch = tgen.add_switch('s1')
534 switch.add_link(tgen.gears['r1'])
536 # Create a connection between r1 and r2
537 switch = tgen.add_switch('s2')
538 switch.add_link(tgen.gears['r1'])
539 switch.add_link(tgen.gears['r2'])
543 Topogen allows us to run the topology without running any tests, you can do
544 that using the following example commands:
548 $ # Running your bootstraped topology
549 $ sudo pytest -s --topology-only new-topo/test_new_topo.py
550 $ # Running the test_template.py topology
551 $ sudo pytest -s --topology-only example-test/test_template.py
552 $ # Running the ospf_topo1.py topology
553 $ sudo pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
555 Parameters explanation:
561 Actives input/output capture. This is required by mininet in order to show
562 the interactive shell.
564 .. option:: --topology-only
566 Don't run any tests, just build the topology.
568 After executing the commands above, you should get the following terminal
573 === test session starts ===
574 platform linux2 -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
575 rootdir: /media/sf_src/topotests, inifile: pytest.ini
578 ospf-topo1/test_ospf_topo1.py *** Starting controller
580 *** Starting 6 switches
581 switch1 switch2 switch3 switch4 switch5 switch6 ...
582 r2: frr zebra started
583 r2: frr ospfd started
584 r3: frr zebra started
585 r3: frr ospfd started
586 r1: frr zebra started
587 r1: frr ospfd started
588 r4: frr zebra started
589 r4: frr ospfd started
593 The last line shows us that we are now using the Mininet CLI (Command Line
594 Interface), from here you can call your router ``vtysh`` or even bash.
596 Here are some commands example:
600 mininet> r1 ping 10.0.3.1
601 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
602 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
603 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
604 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
606 --- 10.0.3.1 ping statistics ---
607 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
608 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
612 mininet> r1 ping 10.0.3.3
613 PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
614 64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.87 ms
615 64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.080 ms
616 64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=0.091 ms
618 --- 10.0.3.3 ping statistics ---
619 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
620 rtt min/avg/max/mdev = 0.080/1.014/2.872/1.313 ms
626 Hello, this is FRRouting (version 3.1-devrzalamena-build).
627 Copyright 1996-2005 Kunihiro Ishiguro, et al.
629 frr-1# show running-config
630 Building configuration...
632 Current configuration:
634 frr version 3.1-devrzalamena-build
635 frr defaults traditional
637 no service integrated-vtysh-config
644 ip address 10.0.3.1/24
647 ip address 10.0.10.1/24
650 ip address 172.16.0.2/24
653 ospf router-id 10.0.255.3
655 redistribute connected
657 network 10.0.3.0/24 area 0
658 network 10.0.10.0/24 area 0
659 network 172.16.0.0/24 area 1
666 After you successfully configured your topology, you can obtain the
667 configuration files (per-daemon) using the following commands:
671 mininet> r3 vtysh -d ospfd
673 Hello, this is FRRouting (version 3.1-devrzalamena-build).
674 Copyright 1996-2005 Kunihiro Ishiguro, et al.
676 frr-1# show running-config
677 Building configuration...
679 Current configuration:
681 frr version 3.1-devrzalamena-build
682 frr defaults traditional
683 no service integrated-vtysh-config
688 ospf router-id 10.0.255.3
690 redistribute connected
692 network 10.0.3.0/24 area 0
693 network 10.0.10.0/24 area 0
694 network 172.16.0.0/24 area 1
704 Test topologies should always be bootstrapped from
705 :file:`tests/topotests/example-test/test_template.py` because it contains
706 important boilerplate code that can't be avoided, like:
708 - imports: os, sys, pytest, topotest/topogen and mininet topology class
709 - The global variable CWD (Current Working directory): which is most likely
710 going to be used to reference the routers configuration file location
716 # For all registered routers, load the zebra configuration file
717 for rname, router in router_list.iteritems():
720 os.path.join(CWD, '{}/zebra.conf'.format(rname))
722 # os.path.join() joins the CWD string with arguments adding the necessary
723 # slashes ('/'). Arguments must not begin with '/'.
725 - The topology class that inherits from Mininet Topo class:
729 class TemplateTopo(Topo):
730 def build(self, *_args, **_opts):
731 tgen = get_topogen(self)
732 # topology build code
734 - pytest ``setup_module()`` and ``teardown_module()`` to start the topology
738 def setup_module(_m):
739 tgen = Topogen(TemplateTopo)
740 tgen.start_topology('debug')
742 def teardown_module(_m):
746 - ``__main__`` initialization code (to support running the script directly)
750 if __name__ == '__main__':
751 sys.exit(pytest.main(["-s"]))
755 - Test code should always be declared inside functions that begin with the
756 ``test_`` prefix. Functions beginning with different prefixes will not be run
758 - Configuration files and long output commands should go into separated files
759 inside folders named after the equipment.
760 - Tests must be able to run without any interaction. To make sure your test
761 conforms with this, run it without the :option:`-s` parameter.
765 - Keep results in stack variables, so people inspecting code with ``pdb`` can
766 easily print their values.
772 assert foobar(router1, router2)
778 result = foobar(router1, router2)
781 - Use ``assert`` messages to indicate where the test failed.
787 for router in router_list:
789 assert condition, 'Router "{}" condition failed'.format(router.name)
794 The most effective ways to inspect topology tests are:
796 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
797 appear when an assertion fails
799 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
801 - Set a breakpoint in the test code with ``pdb``
807 # Add the pdb import at the beginning of the file
811 # Add a breakpoint where you think the problem is
817 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
818 shell allows us to run many useful operations like:
820 - Setting breaking point on file/function/conditions (e.g. ``break``,
822 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
823 - Running python code
827 The TopoGear (equipment abstraction class) implements the ``__str__`` method
828 that allows the user to inspect equipment information.
830 Example of pdb usage:
834 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
835 -> for rnum in range(1, 5):
837 Documented commands (type help <topic>):
838 ========================================
839 EOF bt cont enable jump pp run unt
840 a c continue exit l q s until
841 alias cl d h list quit step up
842 args clear debug help n r tbreak w
843 b commands disable ignore next restart u whatis
844 break condition down j p return unalias where
846 Miscellaneous help topics:
847 ==========================
850 Undocumented commands:
851 ======================
855 116 title2="Expected output")
857 118 def test_ospf_convergence():
858 119 "Test OSPF daemon convergence"
860 121 -> for rnum in range(1, 5):
861 122 router = 'r{}'.format(rnum)
863 124 # Load expected results from the command
864 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
865 126 expected = open(reffile).read()
867 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
868 -> router = 'r{}'.format(rnum)
870 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
871 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
876 (Pdb) tgen = get_topogen()
877 (Pdb) pp tgen.gears[router]
878 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
879 (Pdb) pp str(tgen.gears[router])
880 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
883 121 for rnum in range(1, 5):
884 122 router = 'r{}'.format(rnum)
886 124 # Load expected results from the command
887 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
888 126 expected = open(reffile).read()
890 128 # Run test function until we get an result. Wait at most 60 seconds.
891 129 test_func = partial(compare_show_ip_ospf, router, expected)
892 130 result, diff = topotest.run_and_expect(test_func, '',
893 (Pdb) router1 = tgen.gears[router]
894 (Pdb) router1.vtysh_cmd('show ip ospf route')
895 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
896 (Pdb) tgen.mininet_cli()
900 To enable more debug messages in other Topogen subsystems (like Mininet), more
901 logging messages can be displayed by modifying the test configuration file
907 # Change the default verbosity line from 'info'...
912 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
913 To learn/remember common code snippets see :ref:`topotests-snippets`.
915 Before creating a new topology, make sure that there isn't one already that
916 does what you need. If nothing is similar, then you may create a new topology,
917 preferably, using the newest template
918 (:file:`tests/topotests/example-test/test_template.py`).
920 .. include:: topotests-snippets.rst
925 All the configs and scripts are licensed under a ISC-style license. See Python