Configuring your session

pytest-monitor gives you flexibility for running your test suite. In this section, we will discuss the different available options, and how they influence the pytest session.

Scope Restriction

pytest-monitor is able to restrict the scope of the analysis. As a default, only tests functions discovered by pytest are monitored.

Sometime, you might want to monitor a whole module or test session. This can be achieved thanks to the –restrict-scope-to option.

If a scope restriction is set, then the monitoring will be performed at the selected levels. For example, monitoring at both function and module level can be achieved by the following command:

pytest --restrict-scope-to function,module

Accepted values are:

  • function: test functions will be monitored individually, leading to one entry per test function.

  • module: each discovered module will be monitored regardless of the others.

  • class: test class objects will be monitored individually.

  • session: monitor the whole session.

It is important to realize that using multiple scopes has an impact on the monitoring measures. For example, the pytest-monitor code that monitors functions does consume resources for each function (notably compute time). As a consequence, the resources consumed by their module will include the resources consumed by pytest-monitor for each function. If individual functions were not monitored, the resource consumption reported for the module would therefore be lower.

Due to the way pytest handles test modules, some specificities apply when monitoring modules:

  • The total measured elapsed time includes the setup/teardown process for each function. On the other hand, a function object measures only the duration of the function run (without the setup and teardown parts).

  • Consumed memory will be the peak of memory usage during the whole module run.

Handling parameterized tests

Parameterized tests can be introspected by pytest-monitor during the setup phase: their real name is based on the parameter values. This uses the string representation of the parameters (so you want to make sure that this representation suits your needs).

Let’s consider the following test:

@pytest.mark.parametrize(('asint', 'asstr'), [(10, "10"), (100, "100"), (1000, "1000"), (10000, "10000")])
def test_p(asint, asstr):
    assert asint == int(asstr)

By default, pytest-monitor will generate the following entries:

  • test_p[10-10]

  • test_p[100-100]

  • test_p[1000-1000]

  • test_p[10000-10000]

You can ask pytest-monitor to tag parameters with their names (as provided by @pytest.mark.parametrize), with the following option:

pytest --parametrization-explicit

which will lead to the following entries:

  • test_p[asint_10-asstr_10]

  • test_p[asint_100-asstr_100]

  • test_p[asint_1000-asstr_1000]

  • test_p[asint_10000-asstr_10000]

Disable monitoring

If you need for some reason to disable the monitoring, pass the –no-monitor option.

Describing a run

Sometimes, you might want to compare identical state of your code. In such cases, relying only on the scm references and the run date of the session is not sufficient. For that, pytest-monitor can assist you by tagging your session using description and tags.

Description and tags

The description should be used to provide a brief summary of your run while tags can be used to set special information you want to focus during your analysis. Setting a description is as simple as this:

bash $> pytest --description "Any run description you want"

Flagging your session with specific information is as complex as setting the description:

bash $> pytest --tag pandas=1.0.1 --tag numpy=1.17

This will result in a session with the following description:

{
    "pandas": "1.0.1",
    "numpy": "1.17"
}

You can perfectly use both options to fully describe your session:

bash $> pytest --tag pandas=1.0.1 --tag numpy=1.17 --description "Your summary"

This will result in a session with the following description:

{
    "msg": "Your summary",
    "pandas": "1.0.1",
    "numpy": "1.17"
}

Describing a CI build

For convenience pytest-monitor automatically extends the session’s description with some information extracted from the CI build. For that purpose, pytest-monitor reads the environment at the start of the test session in search for:

  • pipeline_branch, which can either represent a CI pipeline name (preferentially) or the source code branch name.

  • pipeline_build_no, which is the pipeline build number (if available) or the pipeline ID if any.

  • __ci__ which provides you the ci system used.

Currently, pytest-monitor supports the following CI:
  • Gitlab CI

  • Travis CI

  • Jenkins

  • Drone CI

  • Circle CI

The following table explains how both fields are mapped:

CI

pipeline_branch

pipeline_build_no

__ci__

Jenkins CI

BRANCH_NAME if set else JOB_NAME

BUILD_NUMBER

jenkinsci

Drone CI

DRONE_REPO_BRANCH

DRONE_BUILD_NUMBER

droneci

Circle CI

CIRCLE_JOB

CIRCLE_BUILD_NUM

circleci

Gitlab CI

CI_JOB_NAME

CI_PIPELINE_ID

gitlabci

Travis CI

TRAVIS_BUILD_ID

TRAVIS_BUILD_NUMBER

travisci

Note that none of these two fields will be added if:
  • the CI context is incomplete

  • the CI context cannot be computed.

Parameters affecting measures

By default, pytest-monitor runs the garbage collector prior to execute the test function. This leads to finer memory measurements. In the case where you want to disable this call to the garbage collector, you just have to set the option –no-gc on the command line.

bash $> pytest --no-gc