Managing your test suite¶
pytest-monitor does not require any specific setup: it is active by default. Thus all your tests are by default analyzed in order to collect monitored information.
About collecting and storing results¶
pytest-monitor makes a clear distinction between the execution context and the test metrics. This distinction can been seen clearly in the code and the initialization sequence:
Collect environment values. Various pieces of information about the machine are collected.
Store the context. The Execution Context collected in step #1 is recorded if not yet known.
Prepare the run. In order to provide more accurate measurements, we “warm up” the context and take an initial set of measurements. Some will be used for adjusting later measurements.
Run tests and enable measurements. Depending on the item type (function, class or module), we launch the relevant measurements. Each time a monitored item ends, the measurement results (Metrics) are recorded right away.
End session. If sending the monitoring results to a remote server has been requested, this is when pytest-monitor does it.
Selecting tests to monitor¶
By default, all tests are monitored, even small ones which would not require any specific monitoring. It is possible to control more finely which tests will be monitored by pytest-monitor. This is done through the use of pytest markers.
pytest-monitor offers two markers for this:
marks your test for execution, but without any monitoring.
tells pytest-monitor to execute the test but to monitor results if and only if the condition is true.
Here is an example:
import pytest import sys def test_execute_and_monitor(): assert True @pytest.mark.monitor_skip_test def test_execute_do_not_monitor(): assert True @pytest.mark.monitor_skip_test_if(sys.version_info >= (3,)) def test_execute_and_monitor_py3_or_above(): assert True
Disabling monitoring except for some tests¶
pytest offers global markers. For example, one can set the default to no monitoring:
import pytest # With the following global module marker, # monitoring is disabled by default: pytestmark = [pytest.mark.monitor_skip_test]
In this case, it is necessary to explicitly activate individual monitoring. This is accomplished with:
marks your test as to be executed and monitored, even if monitoring is disabled for the module.
tells pytest-monitor to execute the test and to monitor results if and only if the condition is true, regardless of the module monitor setup.
Continuing the example above:
import time import sys def test_executed_not_monitored(): time.sleep(1) assert True def test_executed_not_monitored_2(): time.sleep(2) assert True @pytest.mark.monitor_test def test_executed_and_monitored(): assert True @pytest.mark.monitor_test_if(sys.version_info >= (3, 7)) def test_executed_and_monitored_if_py37(): assert True
Associating your tests to a component¶
pytest-monitor allows you to tag each test in the database with a “component” name. This allows you to identify easily tests that come from a specific part of your application, or for distinguishing test results for two different projects that use the same pytest-monitor database.
Setting up a component name can be done at module level:
import time import pytest pytest_monitor_component = "my_component" # Component name stored in the results database def test_monitored(): t_a = time.time() b_continue = True while b_continue: t_delta = time.time() - t_a b_continue = t_delta < 1 assert not b_continue
If no pytest_monitor_component variable is defined, the component is set to the empty string. In projects with many modules, this can be tedious. pytest-monitor therefore allows you to force a fixed component name for the all the tests:
$ pytest --force-component YOUR_COMPONENT_NAME
This will force the component value to be set to the one you provided, whatever the value of pytest_monitor_component in your test module, if any.
If you need to use a global component name for all your tests while allowing some modules to have a specific component name, you can ask pytest-monitor to add a prefix to any module-level component name:
$ pytest --component-prefix YOUR_COMPONENT_NAME
This way, all tests detected by pytest will have their component prefixed with the given value (tests for modules with no pytest_monitor_component variable are simply tagged with the prefix).
For instance the following test module:
import time import pytest pytest_monitor_component = "component_A" def test_monitored(): t_a = time.time() b_continue = True while b_continue: t_delta = time.time() - t_a b_continue = t_delta < 1 assert not b_continue
will yield the following value for the component fields, depending on the chosen command-line option:
Command line used
pytest –force-component PROJECT_A
pytest –component-prefix PROJECT_A