Preface: the migration to pytest¶
Last year, after 17 years of unittest, I started adopting pytest in psutil (see #2446). The two advantages I cared about most were plain assert statements and pytest-xdist's free parallelism. Tests still inherit from unittest.TestCase and there's no conftest.py or fixture use (rationale in PR-2456).
What I want to focus on here is one of pytest's most frustrating aspects: slow startup.
pytest invocation is slow¶
To measure pytest's startup time, let's run a very simple test where execution time itself is negligible:
$ time python3 -m pytest psutil/tests/test_misc.py::TestMisc::test_version
1 passed in 0.05s
real 0m0,427s
Almost half a second, which is excessive for something I run repeatedly during development. For comparison, the same test under unittest:
$ time python3 -m unittest psutil.tests.test_misc.TestMisc.test_version
Ran 1 test in 0.000s
real 0m0,204s
Roughly twice as fast. Why?
Where is time being spent?¶
A significant portion of pytest's overhead comes from import time, and there's not much one can do about it:
$ time python3 -c "import pytest"
real 0m0,151s
$ time python3 -c "import unittest"
real 0m0,065s
$ time python3 -c "import psutil"
real 0m0,056s
Disable plugin auto loading¶
After some research, I discovered that pytest automatically loads all plugins installed on the system, even if they aren't used. Here's how to list them (output is cut):
$ pytest --trace-config --collect-only
...
active plugins:
...
setupplan : ~/.local/lib/python3.12/site-packages/_pytest/setupplan.py
stepwise : ~/.local/lib/python3.12/site-packages/_pytest/stepwise.py
warnings : ~/.local/lib/python3.12/site-packages/_pytest/warnings.py
logging : ~/.local/lib/python3.12/site-packages/_pytest/logging.py
reports : ~/.local/lib/python3.12/site-packages/_pytest/reports.py
python_path : ~/.local/lib/python3.12/site-packages/_pytest/python_path.py
unraisableexception : ~/.local/lib/python3.12/site-packages/_pytest/unraisableexception.py
threadexception : ~/.local/lib/python3.12/site-packages/_pytest/threadexception.py
faulthandler : ~/.local/lib/python3.12/site-packages/_pytest/faulthandler.py
instafail : ~/.local/lib/python3.12/site-packages/pytest_instafail.py
anyio : ~/.local/lib/python3.12/site-packages/anyio/pytest_plugin.py
pytest_cov : ~/.local/lib/python3.12/site-packages/pytest_cov/plugin.py
subtests : ~/.local/lib/python3.12/site-packages/pytest_subtests/plugin.py
xdist : ~/.local/lib/python3.12/site-packages/xdist/plugin.py
xdist.looponfail : ~/.local/lib/python3.12/site-packages/xdist/looponfail.py
...
It turns out PYTEST_DISABLE_PLUGIN_AUTOLOAD environment variable can be used to disable them. By running PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest --trace-config --collect-only again I can see that the following plugins disappeared:
anyio
pytest_cov
pytest_instafail
pytest_subtests
xdist
xdist.looponfail
Now let's run the test again with PYTEST_DISABLE_PLUGIN_AUTOLOAD:
$ time PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest psutil/tests/test_misc.py::TestMisc::test_version
1 passed in 0.05s
real 0m0,285s
We went from 0.427s to 0.285s, a ~40% improvement. Not bad. We now need to selectively enable only the plugins we actually use, via -p. psutil uses pytest-instafail and pytest-subtests (we'll deal with pytest-xdist later):
$ time PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -p instafail -p subtests ...
real 0m0,320s
Time went back up to 0.320s. Quite a slowdown, but still better than the original 0.427s. Adding pytest-xdist:
$ time PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -p instafail -p subtests -p xdist ...
real 0m0,369s
0.369s. Not much, but still a pity to pay the price when NOT running tests in parallel.
Handling pytest-xdist¶
If we disable pytest-xdist psutil tests still run, but we get a warning:
psutil/tests/test_testutils.py:367
~/svn/psutil/psutil/tests/test_testutils.py:367: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.xdist_group(name="serial")
This warning appears for methods that are intended to run serially, those decorated with @pytest.mark.xdist_group(name="serial"). However, since pytest-xdist is now disabled, the decorator no longer exists. To address this, I implemented the following solution in psutil/tests/__init__.py:
import pytest, functools
PYTEST_PARALLEL = "PYTEST_XDIST_WORKER" in os.environ # True if running parallel tests
if not PYTEST_PARALLEL:
def fake_xdist_group(*_args, **_kwargs):
"""Mimics `@pytest.mark.xdist_group` decorator. No-op: it just
calls the test method or return the decorated class."""
def wrapper(obj):
@functools.wraps(obj)
def inner(*args, **kwargs):
return obj(*args, **kwargs)
return obj if isinstance(obj, type) else inner
return wrapper
pytest.mark.xdist_group = fake_xdist_group # monkey patch
With this in place the warning disappears when running tests serially. To run tests in parallel, we'll manually enable xdist:
$ python3 -m pytest -p xdist -n auto --dist loadgroup
Optimizing test collection time¶
By default, pytest searches the entire directory for tests, adding unnecessary overhead. In pyproject.toml you can tell pytest where test files are located, and only to consider test_*.py files:
[tool.pytest.ini_options]
testpaths = ["psutil/tests/"]
python_files = ["test_*.py"]
Collection time dropped from 0.20s to 0.17s, another ~0.03s shaved off.
Putting it all together¶
With these small optimizations, I managed to reduce pytest startup time by ~0.12 seconds, bringing it down from 0.42 seconds. While this improvement is insignificant for full test runs, it makes a noticeable difference (~28% faster) when repeatedly running individual tests from the command line, which is something I do frequently during development. Final result is visible in PR-2538.