doc/developer/topotests-jsontopo.rst

   1 .. _topotests-json:
   2
   3 Topotests with JSON
   4 ===================
   5
   6 Overview
   7 --------
   8
   9 On top of current topotests framework following enhancements are done:
  10
  11
  12 * Creating the topology and assigning IPs to router' interfaces dynamically.
  13   It is achieved by using json file, in which user specify the number of
  14   routers, links to each router, interfaces for the routers and protocol
  15   configurations for all routers.
  16
  17 * Creating the configurations dynamically. It is achieved by using
  18   :file:`/usr/lib/frr/frr-reload.py` utility, which takes running configuration
  19   and the newly created configuration for any particular router and creates a
  20   delta file(diff file) and loads it to  router.
  21
  22
  23 Logging of test case executions
  24 -------------------------------
  25
  26 * The execution log for each test is saved in the test specific directory create
  27   under `/tmp/topotests` (e.g.,
  28   `/tmp/topotests/<testdirname.testfilename>/exec.log`)
  29
  30 * Additionally all test logs are captured in the `topotest.xml` results file.
  31   This file will be saved in `/tmp/topotests/topotests.xml`. In order to extract
  32   the logs for a particular test one can use the `analyze.py` utility found in
  33   the topotests base directory.
  34
  35 * Router's current configuration, as it is changed during the test, can be
  36   displayed on console or sent to logs by adding ``show_router_config = True`` in
  37   :file:`pytest.ini`.
  38
  39 Note: directory "/tmp/topotests/" is created by topotests by default, making
  40 use of same directory to save execution logs.
  41
  42 Guidelines
  43 ----------
  44
  45 Writing New Tests
  46 ^^^^^^^^^^^^^^^^^
  47
  48 This section will guide you in all recommended steps to produce a standard
  49 topology test.
  50
  51 This is the recommended test writing routine:
  52
  53 * Create a json file which will have routers and protocol configurations
  54 * Write and debug the tests
  55 * Format the new code using `black <https://github.com/psf/black>`_
  56 * Create a Pull Request
  57
  58 .. Note::
  59
  60    BGP tests MUST use generous convergence timeouts - you must ensure that any
  61    test involving BGP uses a convergence timeout that is proportional to the
  62    configured BGP timers. If the timers are not reduced from their defaults this
  63    means 130 seconds; however, it is highly recommended that timers be reduced
  64    from the default values unless the test requires they not be.
  65
  66 File Hierarchy
  67 ^^^^^^^^^^^^^^
  68
  69 Before starting to write any tests one must know the file hierarchy. The
  70 repository hierarchy looks like this:
  71
  72 .. code-block:: console
  73
  74    $ cd frr/tests/topotests
  75    $ find ./*
  76    ...
  77    ./example_test/
  78    ./example_test/test_template_json.json # input json file, having topology, interfaces, bgp and other configuration
  79    ./example_test/test_template_json.py # test script to write and execute testcases
  80    ...
  81    ./lib # shared test/topology functions
  82    ./lib/topojson.py # library to create topology and configurations dynamically from json file
  83    ./lib/common_config.py # library to create protocol's common configurations ex- static_routes, prefix_lists, route_maps etc.
  84    ./lib/bgp.py # library to create and test bgp configurations
  85
  86 Defining the Topology and initial configuration in JSON file
  87 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  88
  89 The first step to write a new test is to define the topology and initial
  90 configuration. User has to define topology and initial configuration in JSON
  91 file. Here is an example of JSON file::
  92
  93    BGP neighborship with single phy-link, sample JSON file:
  94    {
  95    "ipv4base": "192.168.0.0",
  96    "ipv4mask": 30,
  97    "ipv6base": "fd00::",
  98    "ipv6mask": 64,
  99    "link_ip_start": {"ipv4": "192.168.0.0", "v4mask": 30, "ipv6": "fd00::", "v6mask": 64},
 100    "lo_prefix": {"ipv4": "1.0.", "v4mask": 32, "ipv6": "2001:DB8:F::", "v6mask": 128},
 101    "routers": {
 102        "r1": {
 103            "links": {
 104                "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
 105                "r2": {"ipv4": "auto", "ipv6": "auto"},
 106                "r3": {"ipv4": "auto", "ipv6": "auto"}
 107            },
 108            "bgp": {
 109                "local_as": "64512",
 110                "address_family": {
 111                    "ipv4": {
 112                        "unicast": {
 113                            "neighbor": {
 114                                "r2": {
 115                                    "dest_link": {
 116                                        "r1": {}
 117                                    }
 118                                },
 119                                "r3": {
 120                                    "dest_link": {
 121                                        "r1": {}
 122                                    }
 123                                }
 124                            }
 125                        }
 126                    }
 127                }
 128            }
 129        },
 130        "r2": {
 131            "links": {
 132                "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
 133                "r1": {"ipv4": "auto", "ipv6": "auto"},
 134                "r3": {"ipv4": "auto", "ipv6": "auto"}
 135            },
 136            "bgp": {
 137                "local_as": "64512",
 138                "address_family": {
 139                    "ipv4": {
 140                        "unicast": {
 141                            "redistribute": [
 142                                {
 143                                    "redist_type": "static"
 144                                }
 145                            ],
 146                            "neighbor": {
 147                                "r1": {
 148                                    "dest_link": {
 149                                        "r2": {}
 150                                    }
 151                                },
 152                                "r3": {
 153                                    "dest_link": {
 154                                        "r2": {}
 155                                    }
 156                                }
 157                            }
 158                        }
 159                    }
 160                }
 161            }
 162        }
 163        ...
 164
 165
 166 BGP neighboship with loopback interface, sample JSON file::
 167
 168    {
 169    "ipv4base": "192.168.0.0",
 170    "ipv4mask": 30,
 171    "ipv6base": "fd00::",
 172    "ipv6mask": 64,
 173    "link_ip_start": {"ipv4": "192.168.0.0", "v4mask": 30, "ipv6": "fd00::", "v6mask": 64},
 174    "lo_prefix": {"ipv4": "1.0.", "v4mask": 32, "ipv6": "2001:DB8:F::", "v6mask": 128},
 175    "routers": {
 176        "r1": {
 177            "links": {
 178                "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback",
 179                       "add_static_route":"yes"},
 180                "r2": {"ipv4": "auto", "ipv6": "auto"}
 181            },
 182            "bgp": {
 183                "local_as": "64512",
 184                "address_family": {
 185                    "ipv4": {
 186                        "unicast": {
 187                            "neighbor": {
 188                                "r2": {
 189                                    "dest_link": {
 190                                        "lo": {
 191                                            "source_link": "lo"
 192                                        }
 193                                    }
 194                                }
 195                            }
 196                        }
 197                    }
 198                }
 199            },
 200            "static_routes": [
 201                {
 202                    "network": "1.0.2.17/32",
 203                    "next_hop": "192.168.0.1
 204                }
 205            ]
 206        },
 207        "r2": {
 208            "links": {
 209                "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback",
 210                       "add_static_route":"yes"},
 211                "r1": {"ipv4": "auto", "ipv6": "auto"},
 212                "r3": {"ipv4": "auto", "ipv6": "auto"}
 213            },
 214            "bgp": {
 215                "local_as": "64512",
 216                "address_family": {
 217                    "ipv4": {
 218                        "unicast": {
 219                            "redistribute": [
 220                                {
 221                                    "redist_type": "static"
 222                                }
 223                            ],
 224                            "neighbor": {
 225                                "r1": {
 226                                    "dest_link": {
 227                                        "lo": {
 228                                            "source_link": "lo"
 229                                        }
 230                                    }
 231                                },
 232                                "r3": {
 233                                    "dest_link": {
 234                                        "lo": {
 235                                            "source_link": "lo"
 236                                        }
 237                                    }
 238                                }
 239                            }
 240                        }
 241                    }
 242                }
 243            },
 244            "static_routes": [
 245                {
 246                    "network": "192.0.20.1/32",
 247                    "no_of_ip": 9,
 248                    "admin_distance": 100,
 249                    "next_hop": "192.168.0.1",
 250                    "tag": 4001
 251                }
 252            ],
 253        }
 254        ...
 255
 256 BGP neighborship with Multiple phy-links, sample JSON file::
 257
 258    {
 259    "ipv4base": "192.168.0.0",
 260    "ipv4mask": 30,
 261    "ipv6base": "fd00::",
 262    "ipv6mask": 64,
 263    "link_ip_start": {"ipv4": "192.168.0.0", "v4mask": 30, "ipv6": "fd00::", "v6mask": 64},
 264    "lo_prefix": {"ipv4": "1.0.", "v4mask": 32, "ipv6": "2001:DB8:F::", "v6mask": 128},
 265    "routers": {
 266        "r1": {
 267            "links": {
 268                "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
 269                "r2-link1": {"ipv4": "auto", "ipv6": "auto"},
 270                "r2-link2": {"ipv4": "auto", "ipv6": "auto"}
 271            },
 272            "bgp": {
 273                "local_as": "64512",
 274                "address_family": {
 275                    "ipv4": {
 276                        "unicast": {
 277                            "neighbor": {
 278                                "r2": {
 279                                    "dest_link": {
 280                                        "r1-link1": {}
 281                                    }
 282                                }
 283                            }
 284                        }
 285                    }
 286                }
 287            }
 288        },
 289        "r2": {
 290            "links": {
 291                "lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
 292                "r1-link1": {"ipv4": "auto", "ipv6": "auto"},
 293                "r1-link2": {"ipv4": "auto", "ipv6": "auto"},
 294                "r3-link1": {"ipv4": "auto", "ipv6": "auto"},
 295                "r3-link2": {"ipv4": "auto", "ipv6": "auto"}
 296            },
 297            "bgp": {
 298                "local_as": "64512",
 299                "address_family": {
 300                    "ipv4": {
 301                        "unicast": {
 302                            "redistribute": [
 303                                {
 304                                    "redist_type": "static"
 305                                }
 306                            ],
 307                            "neighbor": {
 308                                "r1": {
 309                                    "dest_link": {
 310                                        "r2-link1": {}
 311                                    }
 312                                },
 313                                "r3": {
 314                                    "dest_link": {
 315                                        "r2-link1": {}
 316                                    }
 317                                }
 318                            }
 319                        }
 320                    }
 321                }
 322            }
 323        }
 324        ...
 325
 326
 327 JSON File Explained
 328 """""""""""""""""""
 329
 330 Mandatory keywords/options in JSON:
 331
 332 * ``ipv4base`` : base ipv4 address to generate ips,  ex - 192.168.0.0
 333 * ``ipv4mask`` : mask for ipv4 address, ex - 30
 334 * ``ipv6base`` : base ipv6 address to generate ips,  ex - fd00:
 335 * ``ipv6mask`` : mask for ipv6 address, ex - 64
 336 * ``link_ip_start`` : physical interface base ipv4 and ipv6 address
 337 * ``lo_prefix`` : loopback interface base ipv4 and ipv6 address
 338 * ``routers``   : user can add number of routers as per topology, router's name
 339   can be any logical name, ex- r1 or a0.
 340 * ``r1`` : name of the router
 341 * ``lo`` : loopback interface dict, ipv4 and/or ipv6 addresses generated automatically
 342 * ``type`` : type of interface, to identify loopback interface
 343 * ``links`` : physical interfaces dict, ipv4 and/or ipv6 addresses generated
 344   automatically
 345 * ``r2-link1`` : it will be used when routers have multiple links. 'r2' is router
 346   name, 'link' is any logical name, '1' is to identify link number,
 347   router name and link must be seperated by hyphen (``-``), ex- a0-peer1
 348
 349 Optional keywords/options in JSON:
 350
 351 * ``bgp`` : bgp configuration
 352 * ``local_as`` : Local AS number
 353 * ``unicast`` : All SAFI configuration
 354 * ``neighbor``: All neighbor details
 355 * ``dest_link`` : Destination link to which router will connect
 356 * ``router_id`` : bgp router-id
 357 * ``source_link`` : if user wants to establish bgp neighborship with loopback
 358   interface, add ``source_link``: ``lo``
 359 * ``keepalivetimer`` : Keep alive timer for BGP neighbor
 360 * ``holddowntimer`` : Hold down timer for BGP neighbor
 361 * ``static_routes`` : create static routes for routers
 362 * ``redistribute`` : redistribute static and/or connected routes
 363 * ``prefix_lists`` : create Prefix-lists for routers
 364
 365 Building topology and configurations
 366 """"""""""""""""""""""""""""""""""""
 367
 368 Topology and initial configuration as well as teardown are invoked through the
 369 use of a pytest fixture::
 370
 371
 372    from lib import fixtures
 373
 374    tgen = pytest.fixture(fixtures.tgen_json, scope="module")
 375
 376
 377    # tgen is defined above
 378    # topo is a fixture defined in ../conftest.py and automatically available
 379    def test_bgp_convergence(tgen, topo):
 380        bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
 381        assert bgp_convergence
 382
 383 The `fixtures.topo_json` function calls `topojson.setup_module_from_json()` to
 384 create and return a new `topogen.Topogen()` object using the JSON config file
 385 with the same base filename as the test (i.e., `test_file.py` ->
 386 `test_file.json`). Additionally, the fixture calls `tgen.stop_topology()` after
 387 all the tests have run to cleanup. The function is only invoked once per
 388 file/module (scope="module"), but the resulting object is passed to each
 389 function that has `tgen` as an argument.
 390
 391 For more info on the powerful pytest fixtures feature please see `FIXTURES`_.
 392
 393 .. _FIXTURES: https://docs.pytest.org/en/6.2.x/fixture.html
 394
 395 Creating configuration files
 396 """"""""""""""""""""""""""""
 397
 398 Router's configuration would be saved in config file frr_json.conf. Common
 399 configurations are like, static routes, prefixlists and route maps etc configs,
 400 these configs can be used by any other protocols as it is.
 401 BGP config will be specific to BGP protocol testing.
 402
 403 * json file is passed to API Topogen() which saves the JSON object in
 404   `self.json_topo`
 405 * The Topogen object is then passed to API build_config_from_json(), which looks
 406   for configuration tags in new JSON object.
 407 * If tag is found in the JSON object, configuration is created as per input and
 408   written to file frr_json.conf
 409 * Once JSON parsing is over, frr_json.conf is loaded onto respective router.
 410   Config loading is done using 'vtysh -f <file>'. Initial config at this point
 411   is also saved frr_json_initial.conf. This file can be used to reset
 412   configuration on router, during the course of execution.
 413 * Reset of configuration is done using frr "reload.py" utility, which
 414   calculates the difference between router's running config and user's config
 415   and loads delta file to router. API used - reset_config_on_router()
 416
 417 Writing Tests
 418 """""""""""""
 419
 420 Test topologies should always be bootstrapped from the
 421 `example_test/test_template_json.py` when possible in order to take advantage of
 422 the most recent infrastructure support code.
 423
 424 Example:
 425
 426
 427 * Define a module scoped fixture to setup/teardown and supply the tests with the
 428   `Topogen` object.
 429
 430 .. code-block:: python
 431
 432    import pytest
 433    from lib import fixtures
 434
 435    tgen = pytest.fixture(fixtures.tgen_json, scope="module")
 436
 437
 438 * Define test functions using pytest fixtures
 439
 440 .. code-block:: python
 441
 442    from lib import bgp
 443
 444    # tgen is defined above
 445    # topo is a global available fixture defined in ../conftest.py
 446    def test_bgp_convergence(tgen, topo):
 447        "Test for BGP convergence."
 448
 449        # Don't run this test if we have any failure.
 450        if tgen.routers_have_failure():
 451            pytest.skip(tgen.errors)
 452
 453        bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
 454        assert bgp_convergence