Logging of test case executions
-------------------------------
-* The user can enable logging of testcases execution messages into log file by
- adding ``frrtest_log_dir = /tmp/topotests/`` in :file:`pytest.ini`.
-* Router's current configuration can be displyed on console or sent to logs by
- adding ``show_router_config = True`` in :file:`pytest.ini`.
+* The execution log for each test is saved in the test specific directory create
+ under `/tmp/topotests` (e.g.,
+ `/tmp/topotests/<testdirname.testfilename>/exec.log`)
-Log file name will be displayed when we start execution:
+* Additionally all test logs are captured in the `topotest.xml` results file.
+ This file will be saved in `/tmp/topotests/topotests.xml`. In order to extract
+ the logs for a particular test one can use the `analyze.py` utility found in
+ the topotests base directory.
-.. code-block:: console
-
- root@test:# python ./test_topo_json_single_link.py
-
- Logs will be sent to logfile:
- /tmp/topotests/test_topo_json_single_link_11:57:01.353797
+* Router's current configuration, as it is changed during the test, can be
+ displayed on console or sent to logs by adding ``show_router_config = True`` in
+ :file:`pytest.ini`.
Note: directory "/tmp/topotests/" is created by topotests by default, making
use of same directory to save execution logs.
This is the recommended test writing routine:
-* Create a json file , which will have routers and protocol configurations
-* Create topology from json
-* Create configuration from json
-* Write the tests
+* Create a json file which will have routers and protocol configurations
+* Write and debug the tests
* Format the new code using `black <https://github.com/psf/black>`_
* Create a Pull Request
.. Note::
- BGP tests MUST use generous convergence timeouts - you must ensure
- that any test involving BGP uses a convergence timeout of at least
- 130 seconds.
+ BGP tests MUST use generous convergence timeouts - you must ensure that any
+ test involving BGP uses a convergence timeout that is proportional to the
+ configured BGP timers. If the timers are not reduced from their defaults this
+ means 130 seconds; however, it is highly recommended that timers be reduced
+ from the default values unless the test requires they not be.
File Hierarchy
^^^^^^^^^^^^^^
.. code-block:: console
- $ cd path/to/topotests
+ $ cd frr/tests/topotests
$ find ./*
...
- ./example-topojson-test # the basic example test topology-1
- ./example-topojson-test/test_example_topojson.json # input json file, having
- topology, interfaces, bgp and other configuration
- ./example-topojson-test/test_example_topojson.py # test script to write and
- execute testcases
+ ./example_test/
+ ./example_test/test_template_json.json # input json file, having topology, interfaces, bgp and other configuration
+ ./example_test/test_template_json.py # test script to write and execute testcases
...
./lib # shared test/topology functions
- ./lib/topojson.py # library to create topology and configurations dynamically
- from json file
- ./lib/common_config.py # library to create protocol's common configurations ex-
- static_routes, prefix_lists, route_maps etc.
- ./lib/bgp.py # library to create only bgp configurations
+ ./lib/topojson.py # library to create topology and configurations dynamically from json file
+ ./lib/common_config.py # library to create protocol's common configurations ex- static_routes, prefix_lists, route_maps etc.
+ ./lib/bgp.py # library to create and test bgp configurations
Defining the Topology and initial configuration in JSON file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Building topology and configurations
""""""""""""""""""""""""""""""""""""
-Topology and initial configuration will be created in setup_module(). Following
-is the sample code::
+Topology and initial configuration as well as teardown are invoked through the
+use of a pytest fixture::
- def setup_module(mod):
- json_file = "{}/my_test_name.json".format(CWD)
- tgen = Topogen(json_file, mod.__name__)
- # json topo object is now available in tgen.json_topo
+ from lib import fixtures
- # Starting topology, create tmp files which are loaded to routers
- # to start deamons and then start routers
- start_topology(tgen)
+ tgen = pytest.fixture(fixtures.tgen_json, scope="module")
- # Creating configuration from JSON
- build_config_from_json(tgen)
- def teardown_module(mod):
- tgen = get_topogen()
+ # tgen is defined above
+ # topo is a fixture defined in ../conftest.py and automatically available
+ def test_bgp_convergence(tgen, topo):
+ bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
+ assert bgp_convergence
- # Stop toplogy and Remove tmp files
- stop_topology(tgen)
+The `fixtures.topo_json` function calls `topojson.setup_module_from_json()` to
+create and return a new `topogen.Topogen()` object using the JSON config file
+with the same base filename as the test (i.e., `test_file.py` ->
+`test_file.json`). Additionally, the fixture calls `tgen.stop_topology()` after
+all the tests have run to cleanup. The function is only invoked once per
+file/module (scope="module"), but the resulting object is passed to each
+function that has `tgen` as an argument.
+For more info on the powerful pytest fixtures feature please see `FIXTURES`_.
-* Note: Topology will be created in setup module but routers will not be
- started until we load zebra.conf and bgpd.conf to routers. For all routers
- dirs will be created in /tmp/topotests/<test_folder_name>/<router_name>
- zebra.conf and bgpd.conf empty files will be created and laoded to routers.
- All folder and files are deleted in teardown module..
+.. _FIXTURES: https://docs.pytest.org/en/6.2.x/fixture.html
Creating configuration files
""""""""""""""""""""""""""""
"""""""""""""
Test topologies should always be bootstrapped from the
-example_test/test_template_json.py, because it contains important boilerplate
-code that can't be avoided, like:
-
-imports: os, sys, pytest, and topotest/topogen.
-
-The global variable CWD (Current Working directory): which is most likely going
-to be used to reference the routers configuration file location
+`example_test/test_template_json.py` when possible in order to take advantage of
+the most recent infrastructure support code.
Example:
-* The topology class that inherits from Mininet Topo class;
-
- .. code-block:: python
-
- class TemplateTopo(Topo):
- def build(self, *_args, **_opts):
- tgen = get_topogen(self)
- # topology build code
-
-
-* pytest setup_module() and teardown_module() to start the topology:
+* Define a module scoped fixture to setup/teardown and supply the tests with the
+ `Topogen` object.
- .. code-block:: python
+.. code-block:: python
- def setup_module(_m):
- tgen = Topogen(TemplateTopo)
+ import pytest
+ from lib import fixtures
- # Starting topology, create tmp files which are loaded to routers
- # to start deamons and then start routers
- start_topology(tgen, CWD)
+ tgen = pytest.fixture(fixtures.tgen_json, scope="module")
- def teardown_module(_m):
- tgen = get_topogen()
- # Stop toplogy and Remove tmp files
- stop_topology(tgen, CWD)
+* Define test functions using pytest fixtures
+.. code-block:: python
-* ``__main__`` initialization code (to support running the script directly)
+ from lib import bgp
- .. code-block:: python
+ # tgen is defined above
+ # topo is a global available fixture defined in ../conftest.py
+ def test_bgp_convergence(tgen, topo):
+ "Test for BGP convergence."
- if **name** == '\ **main**\ ':
- sys.exit(pytest.main(["-s"]))
+ # Don't run this test if we have any failure.
+ if tgen.routers_have_failure():
+ pytest.skip(tgen.errors)
+ bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
+ assert bgp_convergence
--- /dev/null
+
+{
+ "address_types": ["ipv4","ipv6"],
+ "ipv4base":"10.0.0.0",
+ "ipv4mask":30,
+ "ipv6base":"fd00::",
+ "ipv6mask":64,
+ "link_ip_start":{"ipv4":"10.0.0.0", "v4mask":30, "ipv6":"fd00::", "v6mask":64},
+ "lo_prefix":{"ipv4":"1.0.", "v4mask":32, "ipv6":"2001:DB8:F::", "v6mask":128},
+ "routers":{
+ "r1":{
+ "links":{
+ "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
+ "r2":{"ipv4":"auto", "ipv6":"auto"},
+ "r3":{"ipv4":"auto", "ipv6":"auto"}
+ },
+ "bgp":{
+ "local_as":"100",
+ "address_family": {
+ "ipv4": {
+ "unicast": {
+ "neighbor": {
+ "r2": {
+ "dest_link": {
+ "r1": {}
+ }
+ },
+ "r3": {
+ "dest_link": {
+ "r1": {}
+ }
+ }
+ }
+ }
+ },
+ "ipv6": {
+ "unicast": {
+ "neighbor": {
+ "r2": {
+ "dest_link": {
+ "r1": {}
+ }
+ },
+ "r3": {
+ "dest_link": {
+ "r1": {}
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "r2":{
+ "links":{
+ "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
+ "r1":{"ipv4":"auto", "ipv6":"auto"},
+ "r3":{"ipv4":"auto", "ipv6":"auto"}
+ },
+ "bgp":{
+ "local_as":"100",
+ "address_family": {
+ "ipv4": {
+ "unicast": {
+ "neighbor": {
+ "r1": {
+ "dest_link": {
+ "r2": {}
+ }
+ },
+ "r3": {
+ "dest_link": {
+ "r2": {}
+ }
+ }
+ }
+ }
+ },
+ "ipv6": {
+ "unicast": {
+ "neighbor": {
+ "r1": {
+ "dest_link": {
+ "r2": {}
+ }
+ },
+ "r3": {
+ "dest_link": {
+ "r2": {}
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "r3":{
+ "links":{
+ "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
+ "r1":{"ipv4":"auto", "ipv6":"auto"},
+ "r2":{"ipv4":"auto", "ipv6":"auto"},
+ "r4":{"ipv4":"auto", "ipv6":"auto"}
+ },
+ "bgp":{
+ "local_as":"100",
+ "address_family": {
+ "ipv4": {
+ "unicast": {
+ "neighbor": {
+ "r1": {
+ "dest_link": {
+ "r3": {}
+ }
+ },
+ "r2": {
+ "dest_link": {
+ "r3": {}
+ }
+ },
+ "r4": {
+ "dest_link": {
+ "r3": {}
+ }
+ }
+ }
+ }
+ },
+ "ipv6": {
+ "unicast": {
+ "neighbor": {
+ "r1": {
+ "dest_link": {
+ "r3": {}
+ }
+ },
+ "r2": {
+ "dest_link": {
+ "r3": {}
+ }
+ },
+ "r4": {
+ "dest_link": {
+ "r3": {}
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "r4":{
+ "links":{
+ "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
+ "r3":{"ipv4":"auto", "ipv6":"auto"}
+ },
+ "bgp":{
+ "local_as":"200",
+ "address_family": {
+ "ipv4": {
+ "unicast": {
+ "neighbor": {
+ "r3": {
+ "dest_link": {
+ "r4": {}
+ }
+ }
+ }
+ }
+ },
+ "ipv6": {
+ "unicast": {
+ "neighbor": {
+ "r3": {
+ "dest_link": {
+ "r4": {}
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+}
--- /dev/null
+#!/usr/bin/env python3
+#
+# September 5 2021, Christian Hopps <chopps@labn.net>
+#
+# Copyright (c) 2021, LabN Consulting, L.L.C.
+# Copyright (c) 2017 by
+# Network Device Education Foundation, Inc. ("NetDEF")
+#
+# Permission to use, copy, modify, and/or distribute this software
+# for any purpose with or without fee is hereby granted, provided
+# that the above copyright notice and this permission notice appear
+# in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND NETDEF DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL NETDEF BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY
+# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
+# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
+# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
+# OF THIS SOFTWARE.
+#
+
+"""
+<template>.py: Test <template>.
+"""
+
+import pytest
+
+# Import topogen and topotest helpers
+from lib import bgp
+from lib import fixtures
+
+
+# TODO: select markers based on daemons used during test
+pytestmark = [
+ pytest.mark.bgpd,
+ # pytest.mark.ospfd,
+ # pytest.mark.ospf6d
+ # ...
+]
+
+# Use tgen_json fixture (invoked by use test arg of same name) to
+# setup/teardown standard JSON topotest
+tgen = pytest.fixture(fixtures.tgen_json, scope="module")
+
+
+# tgen is defined above
+# topo is a fixture defined in ../conftest.py
+def test_bgp_convergence(tgen, topo):
+ "Test for BGP convergence."
+
+ # Don't run this test if we have any failure.
+ if tgen.routers_have_failure():
+ pytest.skip(tgen.errors)
+
+ bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
+ assert bgp_convergence
+
+
+# Memory leak test template
+def test_memory_leak(tgen):
+ "Run the memory leak test and report results."
+
+ if not tgen.is_memleak_enabled():
+ pytest.skip("Memory leak test/report is disabled")
+
+ tgen.report_memory_leaks()
projects / mirror_frr.git / commitdiff