From: Christian Hopps Date: Sun, 5 Sep 2021 06:24:50 +0000 (-0400) Subject: tests: new improved simple JSON template w/ doc update X-Git-Tag: frr-8.2.2~522^2 X-Git-Url: https://git.proxmox.com/?a=commitdiff_plain;h=36c19497301ee10ae952b844dfb4ed016d4bb1e6;p=mirror_frr.git tests: new improved simple JSON template w/ doc update Utilizes new pytest fixtures to completely factor out setup and teardown functionality. Supply the JSON config and write your tests. "The best topotest template yet!" Signed-off-by: Christian Hopps --- diff --git a/doc/developer/topotests-jsontopo.rst b/doc/developer/topotests-jsontopo.rst index 866f27337..e2cc72cc5 100644 --- a/doc/developer/topotests-jsontopo.rst +++ b/doc/developer/topotests-jsontopo.rst @@ -23,19 +23,18 @@ On top of current topotests framework following enhancements are done: Logging of test case executions ------------------------------- -* The user can enable logging of testcases execution messages into log file by - adding ``frrtest_log_dir = /tmp/topotests/`` in :file:`pytest.ini`. -* Router's current configuration can be displyed on console or sent to logs by - adding ``show_router_config = True`` in :file:`pytest.ini`. +* The execution log for each test is saved in the test specific directory create + under `/tmp/topotests` (e.g., + `/tmp/topotests//exec.log`) -Log file name will be displayed when we start execution: +* Additionally all test logs are captured in the `topotest.xml` results file. + This file will be saved in `/tmp/topotests/topotests.xml`. In order to extract + the logs for a particular test one can use the `analyze.py` utility found in + the topotests base directory. -.. code-block:: console - - root@test:# python ./test_topo_json_single_link.py - - Logs will be sent to logfile: - /tmp/topotests/test_topo_json_single_link_11:57:01.353797 +* Router's current configuration, as it is changed during the test, can be + displayed on console or sent to logs by adding ``show_router_config = True`` in + :file:`pytest.ini`. Note: directory "/tmp/topotests/" is created by topotests by default, making use of same directory to save execution logs. @@ -51,18 +50,18 @@ topology test. This is the recommended test writing routine: -* Create a json file , which will have routers and protocol configurations -* Create topology from json -* Create configuration from json -* Write the tests +* Create a json file which will have routers and protocol configurations +* Write and debug the tests * Format the new code using `black `_ * Create a Pull Request .. Note:: - BGP tests MUST use generous convergence timeouts - you must ensure - that any test involving BGP uses a convergence timeout of at least - 130 seconds. + BGP tests MUST use generous convergence timeouts - you must ensure that any + test involving BGP uses a convergence timeout that is proportional to the + configured BGP timers. If the timers are not reduced from their defaults this + means 130 seconds; however, it is highly recommended that timers be reduced + from the default values unless the test requires they not be. File Hierarchy ^^^^^^^^^^^^^^ @@ -72,21 +71,17 @@ repository hierarchy looks like this: .. code-block:: console - $ cd path/to/topotests + $ cd frr/tests/topotests $ find ./* ... - ./example-topojson-test # the basic example test topology-1 - ./example-topojson-test/test_example_topojson.json # input json file, having - topology, interfaces, bgp and other configuration - ./example-topojson-test/test_example_topojson.py # test script to write and - execute testcases + ./example_test/ + ./example_test/test_template_json.json # input json file, having topology, interfaces, bgp and other configuration + ./example_test/test_template_json.py # test script to write and execute testcases ... ./lib # shared test/topology functions - ./lib/topojson.py # library to create topology and configurations dynamically - from json file - ./lib/common_config.py # library to create protocol's common configurations ex- - static_routes, prefix_lists, route_maps etc. - ./lib/bgp.py # library to create only bgp configurations + ./lib/topojson.py # library to create topology and configurations dynamically from json file + ./lib/common_config.py # library to create protocol's common configurations ex- static_routes, prefix_lists, route_maps etc. + ./lib/bgp.py # library to create and test bgp configurations Defining the Topology and initial configuration in JSON file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -370,34 +365,32 @@ Optional keywords/options in JSON: Building topology and configurations """""""""""""""""""""""""""""""""""" -Topology and initial configuration will be created in setup_module(). Following -is the sample code:: +Topology and initial configuration as well as teardown are invoked through the +use of a pytest fixture:: - def setup_module(mod): - json_file = "{}/my_test_name.json".format(CWD) - tgen = Topogen(json_file, mod.__name__) - # json topo object is now available in tgen.json_topo + from lib import fixtures - # Starting topology, create tmp files which are loaded to routers - # to start deamons and then start routers - start_topology(tgen) + tgen = pytest.fixture(fixtures.tgen_json, scope="module") - # Creating configuration from JSON - build_config_from_json(tgen) - def teardown_module(mod): - tgen = get_topogen() + # tgen is defined above + # topo is a fixture defined in ../conftest.py and automatically available + def test_bgp_convergence(tgen, topo): + bgp_convergence = bgp.verify_bgp_convergence(tgen, topo) + assert bgp_convergence - # Stop toplogy and Remove tmp files - stop_topology(tgen) +The `fixtures.topo_json` function calls `topojson.setup_module_from_json()` to +create and return a new `topogen.Topogen()` object using the JSON config file +with the same base filename as the test (i.e., `test_file.py` -> +`test_file.json`). Additionally, the fixture calls `tgen.stop_topology()` after +all the tests have run to cleanup. The function is only invoked once per +file/module (scope="module"), but the resulting object is passed to each +function that has `tgen` as an argument. +For more info on the powerful pytest fixtures feature please see `FIXTURES`_. -* Note: Topology will be created in setup module but routers will not be - started until we load zebra.conf and bgpd.conf to routers. For all routers - dirs will be created in /tmp/topotests// - zebra.conf and bgpd.conf empty files will be created and laoded to routers. - All folder and files are deleted in teardown module.. +.. _FIXTURES: https://docs.pytest.org/en/6.2.x/fixture.html Creating configuration files """""""""""""""""""""""""""" @@ -425,49 +418,37 @@ Writing Tests """"""""""""" Test topologies should always be bootstrapped from the -example_test/test_template_json.py, because it contains important boilerplate -code that can't be avoided, like: - -imports: os, sys, pytest, and topotest/topogen. - -The global variable CWD (Current Working directory): which is most likely going -to be used to reference the routers configuration file location +`example_test/test_template_json.py` when possible in order to take advantage of +the most recent infrastructure support code. Example: -* The topology class that inherits from Mininet Topo class; - - .. code-block:: python - - class TemplateTopo(Topo): - def build(self, *_args, **_opts): - tgen = get_topogen(self) - # topology build code - - -* pytest setup_module() and teardown_module() to start the topology: +* Define a module scoped fixture to setup/teardown and supply the tests with the + `Topogen` object. - .. code-block:: python +.. code-block:: python - def setup_module(_m): - tgen = Topogen(TemplateTopo) + import pytest + from lib import fixtures - # Starting topology, create tmp files which are loaded to routers - # to start deamons and then start routers - start_topology(tgen, CWD) + tgen = pytest.fixture(fixtures.tgen_json, scope="module") - def teardown_module(_m): - tgen = get_topogen() - # Stop toplogy and Remove tmp files - stop_topology(tgen, CWD) +* Define test functions using pytest fixtures +.. code-block:: python -* ``__main__`` initialization code (to support running the script directly) + from lib import bgp - .. code-block:: python + # tgen is defined above + # topo is a global available fixture defined in ../conftest.py + def test_bgp_convergence(tgen, topo): + "Test for BGP convergence." - if **name** == '\ **main**\ ': - sys.exit(pytest.main(["-s"])) + # Don't run this test if we have any failure. + if tgen.routers_have_failure(): + pytest.skip(tgen.errors) + bgp_convergence = bgp.verify_bgp_convergence(tgen, topo) + assert bgp_convergence diff --git a/tests/topotests/example_test/test_example.py b/tests/topotests/example_test/test_example.py index 72eceee61..30c3d248f 100755 --- a/tests/topotests/example_test/test_example.py +++ b/tests/topotests/example_test/test_example.py @@ -36,6 +36,7 @@ def test_fail_example(): assert True, "Some Text with explaination in case of failure" +@pytest.mark.xfail def test_ls_exits_zero(): "Tests for ls command on invalid file" diff --git a/tests/topotests/example_test/test_template_json.json b/tests/topotests/example_test/test_template_json.json new file mode 100644 index 000000000..1ed4a9df6 --- /dev/null +++ b/tests/topotests/example_test/test_template_json.json @@ -0,0 +1,188 @@ + +{ + "address_types": ["ipv4","ipv6"], + "ipv4base":"10.0.0.0", + "ipv4mask":30, + "ipv6base":"fd00::", + "ipv6mask":64, + "link_ip_start":{"ipv4":"10.0.0.0", "v4mask":30, "ipv6":"fd00::", "v6mask":64}, + "lo_prefix":{"ipv4":"1.0.", "v4mask":32, "ipv6":"2001:DB8:F::", "v6mask":128}, + "routers":{ + "r1":{ + "links":{ + "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"}, + "r2":{"ipv4":"auto", "ipv6":"auto"}, + "r3":{"ipv4":"auto", "ipv6":"auto"} + }, + "bgp":{ + "local_as":"100", + "address_family": { + "ipv4": { + "unicast": { + "neighbor": { + "r2": { + "dest_link": { + "r1": {} + } + }, + "r3": { + "dest_link": { + "r1": {} + } + } + } + } + }, + "ipv6": { + "unicast": { + "neighbor": { + "r2": { + "dest_link": { + "r1": {} + } + }, + "r3": { + "dest_link": { + "r1": {} + } + } + } + } + } + } + } + }, + "r2":{ + "links":{ + "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"}, + "r1":{"ipv4":"auto", "ipv6":"auto"}, + "r3":{"ipv4":"auto", "ipv6":"auto"} + }, + "bgp":{ + "local_as":"100", + "address_family": { + "ipv4": { + "unicast": { + "neighbor": { + "r1": { + "dest_link": { + "r2": {} + } + }, + "r3": { + "dest_link": { + "r2": {} + } + } + } + } + }, + "ipv6": { + "unicast": { + "neighbor": { + "r1": { + "dest_link": { + "r2": {} + } + }, + "r3": { + "dest_link": { + "r2": {} + } + } + } + } + } + } + } + }, + "r3":{ + "links":{ + "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"}, + "r1":{"ipv4":"auto", "ipv6":"auto"}, + "r2":{"ipv4":"auto", "ipv6":"auto"}, + "r4":{"ipv4":"auto", "ipv6":"auto"} + }, + "bgp":{ + "local_as":"100", + "address_family": { + "ipv4": { + "unicast": { + "neighbor": { + "r1": { + "dest_link": { + "r3": {} + } + }, + "r2": { + "dest_link": { + "r3": {} + } + }, + "r4": { + "dest_link": { + "r3": {} + } + } + } + } + }, + "ipv6": { + "unicast": { + "neighbor": { + "r1": { + "dest_link": { + "r3": {} + } + }, + "r2": { + "dest_link": { + "r3": {} + } + }, + "r4": { + "dest_link": { + "r3": {} + } + } + } + } + } + } + } + }, + "r4":{ + "links":{ + "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"}, + "r3":{"ipv4":"auto", "ipv6":"auto"} + }, + "bgp":{ + "local_as":"200", + "address_family": { + "ipv4": { + "unicast": { + "neighbor": { + "r3": { + "dest_link": { + "r4": {} + } + } + } + } + }, + "ipv6": { + "unicast": { + "neighbor": { + "r3": { + "dest_link": { + "r4": {} + } + } + } + } + } + } + } + } + } +} diff --git a/tests/topotests/example_test/test_template_json.py b/tests/topotests/example_test/test_template_json.py new file mode 100644 index 000000000..42e8bc6e7 --- /dev/null +++ b/tests/topotests/example_test/test_template_json.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +# +# September 5 2021, Christian Hopps +# +# Copyright (c) 2021, LabN Consulting, L.L.C. +# Copyright (c) 2017 by +# Network Device Education Foundation, Inc. ("NetDEF") +# +# Permission to use, copy, modify, and/or distribute this software +# for any purpose with or without fee is hereby granted, provided +# that the above copyright notice and this permission notice appear +# in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS" AND NETDEF DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL NETDEF BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# + +""" +