Is it possible to run collection sequentially and only execute the tests in parallel?

[mbalty]

We have a test collection mechanism which already uses parallelisation to build some caches in the parent process. These caches should then be accessible from the child processes when running the tests. It seems that pytest-xdist creates the separate processes at the beginning of the pytest run, before collection. Would it be possible to configure it so that it only starts the workers after the tests are collected? We are currently using pytest-parallel which behaves in this manner, but it is not maintained anymore and would like to migrate to pytest-xdist.
Thank you!

[nicoddemus]

Hi @mbalty,

Short answer: it is not. You can see how pytest-xdist works in details here.

[kurt-cb]

I recently abandoned pytest-parallel also, and I used a strategy somewhat like below:

you can simply send the caches to the workers via any RPC mechanism, shared file, or whatever you need. you can use a session scoped fixture that either talks to the parent, or waits for the parent to send them (pull).
You can do this by starting classes in the pytest_cmdline_main hook, then hooking the pytest functions like pytest_runtest and adding your cache init or references. You probably don't need to change xdist to make it work.

@pytest.hookimpl(tryfirst=True)
def pytest_cmdline_main(config):
if getattr(config, "workerinput", None):
# you will get one of these for each worker node
config.pluginmanager.register(XDistWorkerPlugin(config.workerinput), 'myxdistworker')
add_markers(config)
else:
# this is created only on the master node
config.pluginmanager.register(XDistMasterPlugin(options), 'myxdistmaster')