6 Topotests is a suite of topology tests for FRR built on top of Mininet.
11 Only tested with Ubuntu 16.04 and Ubuntu 18.04 (which uses Mininet 2.2.x).
13 Instructions are the same for all setups (i.e. ExaBGP is only used for BGP
16 Installing Mininet Infrastructure
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21 apt-get install mininet
22 apt-get install python-pip
23 apt-get install iproute
25 pip install "pytest<5"
26 pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet
28 useradd -d /var/run/exabgp/ -s /bin/false exabgp
33 Optional, will give better output.
38 disable apport (which move core files)
40 Set ``enabled=0`` in ``/etc/default/apport``.
42 Next, update security limits by changing :file:`/etc/security/limits.conf` to::
44 #<domain> <type> <item> <value>
46 root soft core unlimited
48 root hard core unlimited
50 Reboot for options to take effect.
55 FRR needs to be installed separately. It is assume to be configured like the
56 standard Ubuntu Packages:
58 - Binaries in :file:`/usr/lib/frr`
59 - State Directory :file:`/var/run/frr`
60 - Running under user ``frr``, group ``frr``
61 - vtygroup: ``frrvty``
62 - config directory: :file:`/etc/frr`
63 - For FRR Packages, install the dbg package as well for coredump decoding
65 No FRR config needs to be done and no FRR daemons should be run ahead of the
66 test. They are all started as part of the test.
71 If you prefer to manually build FRR, then use the following suggested config:
77 --localstatedir=/var/run/frr \
78 --sbindir=/usr/lib/frr \
79 --sysconfdir=/etc/frr \
83 --enable-multipath=64 \
86 --enable-vty-group=frrvty \
87 --with-pkg-extra-version=-my-manual-build
89 And create ``frr`` user and ``frrvty`` group as follows:
93 addgroup --system --gid 92 frr
94 addgroup --system --gid 85 frrvty
95 adduser --system --ingroup frr --home /var/run/frr/ \
96 --gecos "FRRouting suite" --shell /bin/false frr
102 Execute all tests with output to console
103 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
107 py.test -s -v --tb=no
109 The above command must be executed from inside the topotests directory.
111 All test\_\* scripts in subdirectories are detected and executed (unless
112 disabled in ``pytest.ini`` file).
114 ``--tb=no`` disables the python traceback which might be irrelevant unless the
115 test script itself is debugged.
125 For example, and assuming you are inside the frr directory:
129 cd tests/topotests/bgp_l3vpn_to_bgp_vrf
130 ./test_bgp_l3vpn_to_bgp_vrf.py
132 For further options, refer to pytest documentation.
134 Test will set exit code which can be used with ``git bisect``.
136 For the simulated topology, see the description in the python file.
138 If you need to clear the mininet setup between tests (if it isn't cleanly
139 shutdown), then use the ``mn -c`` command to clean up the environment.
141 StdErr log from daemos after exit
142 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
144 To enable the reporting of any messages seen on StdErr after the daemons exit,
145 the following env variable can be set::
147 export TOPOTESTS_CHECK_STDERR=Yes
149 (The value doesn't matter at this time. The check is whether the env
150 variable exists or not.) There is no pass/fail on this reporting; the
151 Output will be reported to the console.
153 Collect Memory Leak Information
154 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
156 FRR processes can report unfreed memory allocations upon exit. To
157 enable the reporting of memory leaks, define an environment variable
158 ``TOPOTESTS_CHECK_MEMLEAK`` with the file prefix, i.e.::
160 export TOPOTESTS_CHECK_MEMLEAK="/home/mydir/memleak_"
162 This will enable the check and output to console and the writing of
163 the information to files with the given prefix (followed by testname),
164 ie :file:`/home/mydir/memcheck_test_bgp_multiview_topo1.txt` in case
167 Running Topotests with AddressSanitizer
168 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
170 Topotests can be run with AddressSanitizer. It requires GCC 4.8 or newer.
171 (Ubuntu 16.04 as suggested here is fine with GCC 5 as default). For more
172 information on AddressSanitizer, see
173 https://github.com/google/sanitizers/wiki/AddressSanitizer.
175 The checks are done automatically in the library call of ``checkRouterRunning``
176 (ie at beginning of tests when there is a check for all daemons running). No
177 changes or extra configuration for topotests is required beside compiling the
178 suite with AddressSanitizer enabled.
180 If a daemon crashed, then the errorlog is checked for AddressSanitizer output.
181 If found, then this is added with context (calling test) to
182 :file:`/tmp/AddressSanitizer.txt` in Markdown compatible format.
184 Compiling for GCC AddressSanitizer requires to use ``gcc`` as a linker as well
185 (instead of ``ld``). Here is a suggest way to compile frr with AddressSanitizer
186 for ``master`` branch:
190 git clone https://github.com/FRRouting/frr.git
194 --enable-address-sanitizer \
195 --prefix=/usr/lib/frr --sysconfdir=/etc/frr \
196 --localstatedir=/var/run/frr \
197 --sbindir=/usr/lib/frr --bindir=/usr/lib/frr \
198 --enable-exampledir=/usr/lib/frr/examples \
199 --with-moduledir=/usr/lib/frr/modules \
200 --enable-multipath=0 --enable-rtadv \
201 --enable-tcp-zebra --enable-fpm --enable-pimd \
205 # Create symlink for vtysh, so topotest finds it in /usr/lib/frr
206 sudo ln -s /usr/lib/frr/vtysh /usr/bin/
208 and create ``frr`` user and ``frrvty`` group as shown above.
210 .. _topotests_docker:
212 Running Tests with Docker
213 -------------------------
215 There is a Docker image which allows to run topotests.
220 If you have Docker installed, you can run the topotests in Docker. The easiest
221 way to do this, is to use the make targets from this repository.
223 Your current user needs to have access to the Docker daemon. Alternatively you
224 can run these commands as root.
230 This command will pull the most recent topotests image from Dockerhub, compile
231 FRR inside of it, and run the topotests.
236 Internally, the topotests make target uses a shell script to pull the image and
237 spawn the Docker container.
239 There are several environment variables which can be used to modify the
240 behavior of the script, these can be listed by calling it with ``-h``:
244 ./tests/topotests/docker/frr-topotests.sh -h
246 For example, a volume is used to cache build artifacts between multiple runs of
247 the image. If you need to force a complete recompile, you can set
252 TOPOTEST_CLEAN=1 ./tests/topotests/docker/frr-topotests.sh
254 By default, ``frr-topotests.sh`` will build frr and run pytest. If you append
255 arguments and the first one starts with ``/`` or ``./``, they will replace the
256 call to pytest. If the appended arguments do not match this patttern, they will
257 be provided to pytest as arguments. So, to run a specific test with more
262 ./tests/topotests/docker/frr-topotests.sh -vv -s all-protocol-startup/test_all_protocol_startup.py
264 And to compile FRR but drop into a shell instead of running pytest:
268 ./tests/topotests/docker/frr-topotests.sh /bin/bash
273 The Docker image just includes all the components to run the topotests, but not
274 the topotests themselves. So if you just want to write tests and don't want to
275 make changes to the environment provided by the Docker image. You don't need to
276 build your own Docker image if you do not want to.
278 When developing new tests, there is one caveat though: The startup script of
279 the container will run a ``git-clean`` on its copy of the FRR tree to avoid any
280 pollution of the container with build artefacts from the host. This will also
281 result in your newly written tests being unavailable in the container unless at
282 least added to the index with ``git-add``.
284 If you do want to test changes to the Docker image, you can locally build the
285 image and run the tests without pulling from the registry using the following
291 TOPOTEST_PULL=0 make topotests
294 .. _topotests-guidelines:
302 To run the whole suite of tests the following commands must be executed at the
303 top level directory of topotest:
307 $ # Change to the top level directory of topotests.
308 $ cd path/to/topotests
309 $ # Tests must be run as root, since Mininet requires it.
312 In order to run a specific test, you can use the following command:
316 $ # running a specific topology
317 $ sudo pytest ospf-topo1/
318 $ # or inside the test folder
320 $ sudo pytest # to run all tests inside the directory
321 $ sudo pytest test_ospf_topo1.py # to run a specific test
322 $ # or outside the test folder
324 $ sudo pytest ospf-topo1/test_ospf_topo1.py # to run a specific one
326 The output of the tested daemons will be available at the temporary folder of
331 $ ls /tmp/topotest/ospf-topo1.test_ospf-topo1/r1
333 zebra.err # zebra stderr output
334 zebra.log # zebra log file
335 zebra.out # zebra stdout output
338 You can also run memory leak tests to get reports:
342 $ # Set the environment variable to apply to a specific test...
343 $ sudo env TOPOTESTS_CHECK_MEMLEAK="/tmp/memleak_report_" pytest ospf-topo1/test_ospf_topo1.py
344 $ # ...or apply to all tests adding this line to the configuration file
345 $ echo 'memleak_path = /tmp/memleak_report_' >> pytest.ini
346 $ # You can also use your editor
348 $ # After running tests you should see your files:
349 $ ls /tmp/memleak_report_*
350 memleak_report_test_ospf_topo1.txt
355 This section will guide you in all recommended steps to produce a standard
358 This is the recommended test writing routine:
360 - Write a topology (Graphviz recommended)
361 - Obtain configuration files
362 - Write the test itself
363 - Create a Pull Request
365 Topotest File Hierarchy
366 """""""""""""""""""""""
368 Before starting to write any tests one must know the file hierarchy. The
369 repository hierarchy looks like this:
373 $ cd path/to/topotest
376 ./README.md # repository read me
377 ./GUIDELINES.md # this file
378 ./conftest.py # test hooks - pytest related functions
379 ./example-test # example test folder
380 ./example-test/__init__.py # python package marker - must always exist.
381 ./example-test/test_template.jpg # generated topology picture - see next section
382 ./example-test/test_template.dot # Graphviz dot file
383 ./example-test/test_template.py # the topology plus the test
385 ./ospf-topo1 # the ospf topology test
386 ./ospf-topo1/r1 # router 1 configuration files
387 ./ospf-topo1/r1/zebra.conf # zebra configuration file
388 ./ospf-topo1/r1/ospfd.conf # ospf configuration file
389 ./ospf-topo1/r1/ospfroute.txt # 'show ip ospf' output reference file
390 # removed other for shortness sake
392 ./lib # shared test/topology functions
393 ./lib/topogen.py # topogen implementation
394 ./lib/topotest.py # topotest implementation
396 Guidelines for creating/editing topotest:
398 - New topologies that don't fit the existing directories should create its own
399 - Always remember to add the ``__init__.py`` to new folders, this makes auto
400 complete engines and pylint happy
401 - Router (Quagga/FRR) specific code should go on topotest.py
402 - Generic/repeated router actions should have an abstraction in
404 - Generic/repeated non-router code should go to topotest.py
405 - pytest related code should go to conftest.py (e.g. specialized asserts)
407 Defining the Topology
408 """""""""""""""""""""
410 The first step to write a new test is to define the topology. This step can be
411 done in many ways, but the recommended is to use Graphviz to generate a drawing
412 of the topology. It allows us to see the topology graphically and to see the
413 names of equipment, links and addresses.
415 Here is an example of Graphviz dot file that generates the template topology
416 :file:`tests/topotests/example-test/test_template.dot` (the inlined code might
417 get outdated, please see the linked file)::
439 label="s1\n192.168.0.0/24",
445 label="s2\n192.168.1.0/24",
451 r1 -- s1 [label="eth0\n.1"];
453 r1 -- s2 [label="eth1\n.100"];
454 r2 -- s2 [label="eth0\n.1"];
457 Here is the produced graph:
481 label="s1\n192.168.0.0/24",
487 label="s2\n192.168.1.0/24",
493 r1 -- s1 [label="eth0\n.1"];
495 r1 -- s2 [label="eth1\n.100"];
496 r2 -- s2 [label="eth0\n.1"];
499 Generating / Obtaining Configuration Files
500 """"""""""""""""""""""""""""""""""""""""""
502 In order to get the configuration files or command output for each router, we
503 need to run the topology and execute commands in ``vtysh``. The quickest way to
504 achieve that is writing the topology building code and running the topology.
506 To bootstrap your test topology, do the following steps:
508 - Copy the template test
513 $ touch new-topo/__init__.py
514 $ cp example-test/test_template.py new-topo/test_new_topo.py
516 - Modify the template according to your dot file
518 Here is the template topology described in the previous section in python code:
522 class TemplateTopo(Topo):
523 "Test topology builder"
524 def build(self, *_args, **_opts):
526 tgen = get_topogen(self)
529 for routern in range(1, 3):
530 tgen.add_router('r{}'.format(routern))
532 # Create a switch with just one router connected to it to simulate a
534 switch = tgen.add_switch('s1')
535 switch.add_link(tgen.gears['r1'])
537 # Create a connection between r1 and r2
538 switch = tgen.add_switch('s2')
539 switch.add_link(tgen.gears['r1'])
540 switch.add_link(tgen.gears['r2'])
544 Topogen allows us to run the topology without running any tests, you can do
545 that using the following example commands:
549 $ # Running your bootstraped topology
550 $ sudo pytest -s --topology-only new-topo/test_new_topo.py
551 $ # Running the test_template.py topology
552 $ sudo pytest -s --topology-only example-test/test_template.py
553 $ # Running the ospf_topo1.py topology
554 $ sudo pytest -s --topology-only ospf-topo1/test_ospf_topo1.py
556 Parameters explanation:
562 Actives input/output capture. This is required by mininet in order to show
563 the interactive shell.
565 .. option:: --topology-only
567 Don't run any tests, just build the topology.
569 After executing the commands above, you should get the following terminal
574 === test session starts ===
575 platform linux2 -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
576 rootdir: /media/sf_src/topotests, inifile: pytest.ini
579 ospf-topo1/test_ospf_topo1.py *** Starting controller
581 *** Starting 6 switches
582 switch1 switch2 switch3 switch4 switch5 switch6 ...
583 r2: frr zebra started
584 r2: frr ospfd started
585 r3: frr zebra started
586 r3: frr ospfd started
587 r1: frr zebra started
588 r1: frr ospfd started
589 r4: frr zebra started
590 r4: frr ospfd started
594 The last line shows us that we are now using the Mininet CLI (Command Line
595 Interface), from here you can call your router ``vtysh`` or even bash.
597 Here are some commands example:
601 mininet> r1 ping 10.0.3.1
602 PING 10.0.3.1 (10.0.3.1) 56(84) bytes of data.
603 64 bytes from 10.0.3.1: icmp_seq=1 ttl=64 time=0.576 ms
604 64 bytes from 10.0.3.1: icmp_seq=2 ttl=64 time=0.083 ms
605 64 bytes from 10.0.3.1: icmp_seq=3 ttl=64 time=0.088 ms
607 --- 10.0.3.1 ping statistics ---
608 3 packets transmitted, 3 received, 0% packet loss, time 1998ms
609 rtt min/avg/max/mdev = 0.083/0.249/0.576/0.231 ms
613 mininet> r1 ping 10.0.3.3
614 PING 10.0.3.3 (10.0.3.3) 56(84) bytes of data.
615 64 bytes from 10.0.3.3: icmp_seq=1 ttl=64 time=2.87 ms
616 64 bytes from 10.0.3.3: icmp_seq=2 ttl=64 time=0.080 ms
617 64 bytes from 10.0.3.3: icmp_seq=3 ttl=64 time=0.091 ms
619 --- 10.0.3.3 ping statistics ---
620 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
621 rtt min/avg/max/mdev = 0.080/1.014/2.872/1.313 ms
627 Hello, this is FRRouting (version 3.1-devrzalamena-build).
628 Copyright 1996-2005 Kunihiro Ishiguro, et al.
630 frr-1# show running-config
631 Building configuration...
633 Current configuration:
635 frr version 3.1-devrzalamena-build
636 frr defaults traditional
638 no service integrated-vtysh-config
645 ip address 10.0.3.1/24
648 ip address 10.0.10.1/24
651 ip address 172.16.0.2/24
654 ospf router-id 10.0.255.3
656 redistribute connected
658 network 10.0.3.0/24 area 0
659 network 10.0.10.0/24 area 0
660 network 172.16.0.0/24 area 1
667 After you successfully configured your topology, you can obtain the
668 configuration files (per-daemon) using the following commands:
672 mininet> r3 vtysh -d ospfd
674 Hello, this is FRRouting (version 3.1-devrzalamena-build).
675 Copyright 1996-2005 Kunihiro Ishiguro, et al.
677 frr-1# show running-config
678 Building configuration...
680 Current configuration:
682 frr version 3.1-devrzalamena-build
683 frr defaults traditional
684 no service integrated-vtysh-config
689 ospf router-id 10.0.255.3
691 redistribute connected
693 network 10.0.3.0/24 area 0
694 network 10.0.10.0/24 area 0
695 network 172.16.0.0/24 area 1
705 Test topologies should always be bootstrapped from
706 :file:`tests/topotests/example-test/test_template.py` because it contains
707 important boilerplate code that can't be avoided, like:
709 - imports: os, sys, pytest, topotest/topogen and mininet topology class
710 - The global variable CWD (Current Working directory): which is most likely
711 going to be used to reference the routers configuration file location
717 # For all registered routers, load the zebra configuration file
718 for rname, router in router_list.iteritems():
721 os.path.join(CWD, '{}/zebra.conf'.format(rname))
723 # os.path.join() joins the CWD string with arguments adding the necessary
724 # slashes ('/'). Arguments must not begin with '/'.
726 - The topology class that inherits from Mininet Topo class:
730 class TemplateTopo(Topo):
731 def build(self, *_args, **_opts):
732 tgen = get_topogen(self)
733 # topology build code
735 - pytest ``setup_module()`` and ``teardown_module()`` to start the topology
739 def setup_module(_m):
740 tgen = Topogen(TemplateTopo)
741 tgen.start_topology('debug')
743 def teardown_module(_m):
747 - ``__main__`` initialization code (to support running the script directly)
751 if __name__ == '__main__':
752 sys.exit(pytest.main(["-s"]))
756 - Test code should always be declared inside functions that begin with the
757 ``test_`` prefix. Functions beginning with different prefixes will not be run
759 - Configuration files and long output commands should go into separated files
760 inside folders named after the equipment.
761 - Tests must be able to run without any interaction. To make sure your test
762 conforms with this, run it without the :option:`-s` parameter.
766 - Keep results in stack variables, so people inspecting code with ``pdb`` can
767 easily print their values.
773 assert foobar(router1, router2)
779 result = foobar(router1, router2)
782 - Use ``assert`` messages to indicate where the test failed.
788 for router in router_list:
790 assert condition, 'Router "{}" condition failed'.format(router.name)
795 The most effective ways to inspect topology tests are:
797 - Run pytest with ``--pdb`` option. This option will cause a pdb shell to
798 appear when an assertion fails
800 Example: ``pytest -s --pdb ospf-topo1/test_ospf_topo1.py``
802 - Set a breakpoint in the test code with ``pdb``
808 # Add the pdb import at the beginning of the file
812 # Add a breakpoint where you think the problem is
818 The `Python Debugger <https://docs.python.org/2.7/library/pdb.html>`__ (pdb)
819 shell allows us to run many useful operations like:
821 - Setting breaking point on file/function/conditions (e.g. ``break``,
823 - Inspecting variables (e.g. ``p`` (print), ``pp`` (pretty print))
824 - Running python code
828 The TopoGear (equipment abstraction class) implements the ``__str__`` method
829 that allows the user to inspect equipment information.
831 Example of pdb usage:
835 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(121)test_ospf_convergence()
836 -> for rnum in range(1, 5):
838 Documented commands (type help <topic>):
839 ========================================
840 EOF bt cont enable jump pp run unt
841 a c continue exit l q s until
842 alias cl d h list quit step up
843 args clear debug help n r tbreak w
844 b commands disable ignore next restart u whatis
845 break condition down j p return unalias where
847 Miscellaneous help topics:
848 ==========================
851 Undocumented commands:
852 ======================
856 116 title2="Expected output")
858 118 def test_ospf_convergence():
859 119 "Test OSPF daemon convergence"
861 121 -> for rnum in range(1, 5):
862 122 router = 'r{}'.format(rnum)
864 124 # Load expected results from the command
865 125 reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
866 126 expected = open(reffile).read()
868 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(122)test_ospf_convergence()
869 -> router = 'r{}'.format(rnum)
871 > /media/sf_src/topotests/ospf-topo1/test_ospf_topo1.py(125)test_ospf_convergence()
872 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
877 (Pdb) tgen = get_topogen()
878 (Pdb) pp tgen.gears[router]
879 <lib.topogen.TopoRouter object at 0x7f74e06c9850>
880 (Pdb) pp str(tgen.gears[router])
881 'TopoGear<name="r1",links=["r1-eth0"<->"s1-eth0","r1-eth1"<->"s3-eth0"]> TopoRouter<>'
884 121 for rnum in range(1, 5):
885 122 router = 'r{}'.format(rnum)
887 124 # Load expected results from the command
888 125 -> reffile = os.path.join(CWD, '{}/ospfroute.txt'.format(router))
889 126 expected = open(reffile).read()
891 128 # Run test function until we get an result. Wait at most 60 seconds.
892 129 test_func = partial(compare_show_ip_ospf, router, expected)
893 130 result, diff = topotest.run_and_expect(test_func, '',
894 (Pdb) router1 = tgen.gears[router]
895 (Pdb) router1.vtysh_cmd('show ip ospf route')
896 '============ OSPF network routing table ============\r\nN 10.0.1.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth0\r\nN 10.0.2.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.3, r1-eth1\r\nN 10.0.3.0/24 [10] area: 0.0.0.0\r\n directly attached to r1-eth1\r\nN 10.0.10.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.0.0/24 [20] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\nN IA 172.16.1.0/24 [30] area: 0.0.0.0\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF router routing table =============\r\nR 10.0.255.2 [10] area: 0.0.0.0, ASBR\r\n via 10.0.3.3, r1-eth1\r\nR 10.0.255.3 [10] area: 0.0.0.0, ABR, ASBR\r\n via 10.0.3.1, r1-eth1\r\nR 10.0.255.4 IA [20] area: 0.0.0.0, ASBR\r\n via 10.0.3.1, r1-eth1\r\n\r\n============ OSPF external routing table ===========\r\n\r\n\r\n'
897 (Pdb) tgen.mininet_cli()
901 To enable more debug messages in other Topogen subsystems (like Mininet), more
902 logging messages can be displayed by modifying the test configuration file
908 # Change the default verbosity line from 'info'...
913 Instructions for use, write or debug topologies can be found in :ref:`topotests-guidelines`.
914 To learn/remember common code snippets see :ref:`topotests-snippets`.
916 Before creating a new topology, make sure that there isn't one already that
917 does what you need. If nothing is similar, then you may create a new topology,
918 preferably, using the newest template
919 (:file:`tests/topotests/example-test/test_template.py`).
921 .. include:: topotests-snippets.rst
926 All the configs and scripts are licensed under a ISC-style license. See Python