Testing

This guide includes:

By following the steps here, you should have a virtual environment and a clone of connect module locally.

Constructing A LAB File

All the tests are parametrized so that multiple developers can execute them on the same LAB without interfering with each other. The user specific data is presented in the LAB file, under tests/labs/user/ directory.

To select a LAB when running the tests, use the --lab option, specifying in argument the name of the file, except the .yaml extension. All LAB files should end with the .yaml extension.

Example

Here is an example of the LAB file, you can copy it in this directory, and modify the values.

environment:
  id: 8404deea-3621-4bca-9076-6a612115f810 # Change this
  name: ci-1 # Change this
  project:
    id: 3fa85f64-5717-4562-b3fc-2c963f66afa6
    name: connect

prefix: ci-1 # Change this

service_id_range:
  start: 10000 # Change this
  stop: 10099 # Change this

vlan_id_range:
  start: 600 # Change this
  stop: 649 # Change this

LAB Topology Constraints

In the test cases some assumptions are made on the structure of the lab. Those assumptions are checked when the tests using the LAB manager are setup. Here are the assumptions:

  1. The topology file contains at least four subscribers, named: subscriber-1, subscriber-2, subscriber-3 and subscriber-4.

  2. Side A (subscriber-1 and subscriber-2) are connected to the same provider router.

  3. Side B (subscriber-3 and subscriber-4) are connected to the same provider router.

  4. Side A and B do not have a direct connection

  5. All routers in the LAB have the same vendor.

Here is a visual representation of the topology:

Topology

Vendor specific tests

Some tests can only be executed with a specific topology file because they test the behavior of the module against a specific vendor. To avoid running this test case in unsupported configurations, pytest’s markers can be used.

Example

This is already the case for two test cases:

  • tests/test_model/test_carrier_ethernet_evc_nokia/test_LDP

  • tests/test_model/test_carrier_ethernet_evc_nokia/test_EVPN

These verify that the generated yang config contains the desired information. In order to mark those tests as suitable only for Nokia topology, they get marked with the marker nokia_only:

@pytest.mark.nokia_only
def test_LDP(project: Project, lab_config: LabConfig) -> None:

This marker can be reused for any test that requires a Nokia topology file.

By default those tests will be skipped. If you are running the tests with a topology file containing only Nokia devices, you can either set the environment variable INMANTA_CONNECT_LAB_KIND or the cli parameter --connect-lab-kind to Nokia. In that case, and in that case only, those two tests will be executed.

Adding New Markers

To add some additional markers, i.e. when other vendor-specific tests are added apart from Nokia, there are three things to do:

  1. Extend the pytest_configure function in tests/conftest.py to support the additional marker.

  2. Extend the pytest_runtest_setup function in tests/conftest.py to skip the test if the marker is detected but the vendor is not the one we want.

  3. Add the marker on top of the test you wish to limit to the specific vendor.

Topology File Structure

The topology file should contain four main entries:

  • routers: A dictionary containing the routers of the LAB.

  • subscribers: A dictionary containing the subscribers of the LAB.

  • links: A list containing the links between different element (routers and subscribers) in the LAB.

  • inventory: A dictionary containing a valid inventory that can be given to the module for this LAB.

Here is an example of the topology file:

routers:
  nokia-east:
    vendor: Nokia
    mgmt_address: 192.168.2.33
    netconf_port: 20830
    ssh_port: 20022
    mgmt_username_env: NETCONF_DEVICE_USER
    mgmt_password_env: NETCONF_DEVICE_PASSWORD

  nokia-west:
    vendor: Nokia
    mgmt_address: 192.168.2.33
    netconf_port: 21830
    ssh_port: 21022
    mgmt_username_env: NETCONF_DEVICE_USER
    mgmt_password_env: NETCONF_DEVICE_PASSWORD

subscribers:
  subscriber-1:
    mgmt_address: 192.168.2.33
    api_port: 2001
    uni: inmanta:456-852-789
  subscriber-2:
    mgmt_address: 192.168.2.33
    api_port: 2002
    uni: inmanta:456-985-752
  subscriber-3:
    mgmt_address: 192.168.2.33
    api_port: 2003
    uni: inmanta:123-852-456
  subscriber-4:
    mgmt_address: 192.168.2.33
    api_port: 2004
    uni: inmanta:652-784-963

links:
  - endpoints:
    - type: router
      device: nokia-east
      interface: eth1
    - type: subscriber
      device: subscriber-1
      interface: eth1
      namespace: east1
  - endpoints:
    - type: router
      device: nokia-east
      interface: eth2
    - type: subscriber
      device: subscriber-2
      interface: eth1
      namespace: east1
  - endpoints:
    - type: router
      device: nokia-west
      interface: eth1
    - type: subscriber
      device: subscriber-3
      interface: eth1
      namespace: west1
  - endpoints:
    - type: router
      device: nokia-west
      interface: eth2
    - type: subscriber
      device: subscriber-4
      interface: eth1
      namespace: west1

inventory:
  devices:
    dev-1:
      mgmt_ip: 172.20.20.31
      mgmt_port: 22
      vendor: Nokia
      model: 7750 SR
      os: TiMos
      version: "20.10"
      username_env: NETCONF_DEVICE_USER
      password_env: NETCONF_DEVICE_PASSWORD
    dev-2:
      mgmt_ip: 172.20.20.21
      mgmt_port: 22
      vendor: Nokia
      model: 7750 SR
      os: TiMos
      version: "20.10"
      username_env: NETCONF_DEVICE_USER
      password_env: NETCONF_DEVICE_PASSWORD

  ne:
    - id: "1"
      name: Nokia_west
      router_ip: 10.255.255.2
      device: ref#devices.dev-1
    - id: "2"
      name: Nokia_east
      router_ip: 10.255.255.1
      device: ref#devices.dev-2

  uni:
    - id: inmanta:456-985-752
      port: 1/1/c3/1
      network_element: ref#ne[id=1]
    - id: inmanta:456-852-789
      port: 1/1/c2/1
      network_element: ref#ne[id=1]
    - id: inmanta:652-784-963
      port: 1/1/c3/1
      network_element: ref#ne[id=2]
    - id: inmanta:123-852-456
      port: 1/1/c2/1
      network_element: ref#ne[id=2]

Here are some notes about parametrized values:

  • Some values in the file can be parametrized using the prefix opt: prefix.

  • The remaining part of the value will be extracted and resolved.

  • The remaining value should match the key of one of the TestParameter object set in the tests conftest.py file.

For instance, if you set opt:inm_con_lab_mgmt_ip as a value for routers.nokia-east.mgmt_address, when the topology file is loaded, this value will be replaced by the value passed to the --lab-mgmt-ip argument or set in INMANTA_CONNECT_LAB_MGMT_IP. Please note that CLI arguments have more priority than env variables.

Assembling A Basic Test

We also need to make a temporary directory for the tests and utilize it by environment variables:

mkdir /tmp/env
export INMANTA_TEST_ENV=/tmp/env

The most basic type of testing is a compile test which can be conducted like:

import uuid
import pytest
from pytest_inmanta.plugin import Project

def test_basics(project: Project) -> None:
    """
    Simple example of instantiation of a CarrierEthernetEvc entity.  We just try to compile it.
    """
    model = f"""
        import connect
        import connect::infra as infra
        import connect::mock

        connect::CarrierEthernetEvc(
            identifier="my-evc-001",
            connectionType="POINT_TO_POINT",
            ext_networkBackend=connect::Ext_networkBackend(type="EVPN"),
            evcEndPoints=[
                connect::CarrierEthernetEvcEndPoint(
                    identifier="my-evc-ep-1",
                    egressBandwidthProfilePerEndPoint=[
                        connect::EgressBwpFlow(
                            cir=1,
                        ),
                        connect::EgressBwpFlow(
                            cir=2,
                        ),
                    ],
                    evcEndPointMap=connect::VlanIdListOrUntag(
                        type="LIST",
                        vlanIdList=[
                            connect::VlanId(vlanId=200),
                        ],
                    ),
                    carrierEthernetSubscriberUni=connect::CarrierEthernetSubscriberUniRef(
                        href="inmanta:456-985-752",
                    ),
                    _uni=infra::UserNetworkInterface(
                        id="inmanta:456-985-752",
                        port="1/1/c3/1",
                        network_element=infra::NetworkElement(
                            id="1",
                            name="ne-1",
                            router_ip="1.2.3.4",
                            device=infra::Device(
                                mgmt_ip="1.2.3.4",
                                mgmt_port=22,
                                vendor="Nokia",
                                model="7750 SR",
                                os="TiMos",
                                version="20.10",
                                username_env="NETCONF_DEVICE_USER",
                                password_env="NETCONF_DEVICE_PASSWORD",
                            )
                        )
                    ),
                ),
                connect::CarrierEthernetEvcEndPoint(
                    identifier="my-evc-ep-2",
                    egressBandwidthProfilePerEndPoint=[
                        connect::EgressBwpFlow(
                            cir=1,
                        ),
                        connect::EgressBwpFlow(
                            cir=2,
                        ),
                    ],
                    evcEndPointMap=connect::VlanIdListOrUntag(
                        type="LIST",
                        vlanIdList=[
                            connect::VlanId(vlanId=201),
                        ],
                    ),
                    carrierEthernetSubscriberUni=connect::CarrierEthernetSubscriberUniRef(
                        href="inmanta:652-784-963",
                    ),
                    _uni=infra::UserNetworkInterface(
                        id="inmanta:652-784-963",
                        port="1/1/c3/1",
                        network_element=infra::NetworkElement(
                            id="2",
                            name="ne-2",
                            router_ip="1.2.3.5",
                            device=infra::Device(
                                mgmt_ip="1.2.3.5",
                                mgmt_port=22,
                                vendor="Nokia",
                                model="7750 SR",
                                os="TiMos",
                                version="20.10",
                                username_env="NETCONF_DEVICE_USER",
                                password_env="NETCONF_DEVICE_PASSWORD",
                            )
                        )
                    ),
                ),
            ],
            purge_resources=false,
            instance_id="{uuid.uuid4()}",
        )
    """

    project.compile(model, no_dedent=False)

Running Tests

# There are multiple env files with the names of vendors. These match a specific topology file 
# in tests/labs/topology
# Sourcing one of this files will also run the common-env.sh script. This script loads some 
# environment variables common to any developer of the module.
# If you wish to load some env variables automatically, this script looks for the existence
# of personal_env.sh file and executes it, if found. You can add all private envs you require there.
source <topology>-env.sh
export INMANTA_CONNECT_LAB_USER="" # The name of the lab file you created (if not added to personal_env.sh)
export INMANTA_LSM_ENVIRONMENT=""  # An environment on mentioned orchestrator (if not added to personal_env.sh)

pytest tests \
    --use-module-in-place \
    --log-cli-level=debug \
    tests/

Running Mypy Type Check

The use of Mypy is encouraged since it makes the code easier to read and debug should a problem occur. Mypy is an optional static type checker for Python that aims to combine the benefits of dynamic or "duck" typing and static typing.

# Check typing in the plugins
make mypy-plugins

# Check typing in the tests
make mypy-tests

# Check typing in both plugins and tests
make mypy

To see if you improved the typing, do make mypy-save before you make changes and make mypy-diff afterwards