¿Hay alguna forma de controlar cómo pytest-xdist ejecuta las pruebas en paralelo?

I have the following directory layout:

runner.py
lib/
tests/
      testsuite1/
                 testsuite1.py
      testsuite2/
                 testsuite2.py
      testsuite3/
                 testsuite3.py
      testsuite4/
                 testsuite4.py

The format of testsuite*.py modules is as follows:

import pytest 
class testsomething:
      def setup_class(self):
          ''' do some setup '''
          # Do some setup stuff here      
      def teardown_class(self):
          '''' do some teardown'''
          # Do some teardown stuff here

      def test1(self):
          # Do some test1 related stuff

      def test2(self):
          # Do some test2 related stuff

      ....
      ....
      ....
      def test40(self):
          # Do some test40 related stuff

if __name__=='__main()__'
   pytest.main(args=[os.path.abspath(__file__)])

The problem I have is that I would like to execute the 'testsuites' in parallel i.e. I want testsuite1, testsuite2, testsuite3 and testsuite4 to start execution in parallel but individual tests within the testsuites need to be executed serially.

When I use the 'xdist' plugin from py.test and kick off the tests using 'py.test -n 4', py.test is gathering all the tests and randomly load balancing the tests among 4 workers. This leads to the 'setup_class' method to be executed every time of each test within a 'testsuitex.py' module (which defeats my purpose. I want setup_class to be executed only once per class and tests executed serially there after).

Essentially what I want the execution to look like is:

worker1: executes all tests in testsuite1.py serially
worker2: executes all tests in testsuite2.py serially
worker3: executes all tests in testsuite3.py serially
worker4: executes all tests in testsuite4.py serially

mientras worker1, worker2, worker3 and worker4 are all executed in parallel.

Is there a way to achieve this in 'pytest-xidst' framework?

The only option that I can think of is to kick off different processes to execute each test suite individually within runner.py:

def test_execute_func(testsuite_path):
    subprocess.process('py.test %s' % testsuite_path)

if __name__=='__main__':
   #Gather all the testsuite names
   for each testsuite:
       multiprocessing.Process(test_execute_func,(testsuite_path,))

preguntado el 09 de enero de 11 a las 00:01

5 Respuestas

Puedes usar --dist=loadscope to group all the tests in the same test class. Here is the doc from pytest-xdist on pypi

By default, the -n option will send pending tests to any worker that is available, without any guaranteed order, but you can control this with these options:

--dist=loadscope: tests will be grouped by module for test functions and by class for test methods, then each group will be sent to an available worker, guaranteeing that all tests in a group run in the same process. This can be useful if you have expensive module-level or class-level fixtures. Currently the groupings can’t be customized, with grouping by class takes priority over grouping by module. This feature was added in version 1.19.

--dist=loadfile: tests will be grouped by file name, and then will be sent to an available worker, guaranteeing that all tests in a group run in the same worker. This feature was added in version 1.21.

Respondido el 06 de junio de 19 a las 12:06

Múltiples opciones disponibles

Yes, there are such ways, the available options per xdist version 1.28.0 is the following ones:

  • --dist=each: Sends all the tests to all the nodes, so each test is run on every node.
  • --dist=load: Distributes the tests collected across all nodes so each test is run just once. All nodes collect and submit the test suite and when all collections (of test suites) are received it is verified they are identical collections. Then the collection gets divided up in chunks and chunks get submitted to nodes for execution.
  • --dist=loadscope Distributes the tests collected across all nodes so each test is run just once. All nodes collect and submit the list of tests and when all collections are received it is verified they are identical collections. Then the collection gets divided up in work units, grouped by test scope, and those work units get submitted to nodes.
  • --dist=loadfile Distributes the tests collected across all nodes so each test is run just once. All nodes collect and submit the list of tests and when all collections are received it is verified they are identical collections. Then the collection gets divided up in work units, grouped by test file, and those work units get submitted to nodes.

If you need any further information I recommend you to go straightly into the actual implementation of the schedulers and checkout how the distribution is being done.

Respondido 08 Feb 21, 19:02

I was wondering how does -n and ` --forked` or --boxed options relate to the the --dist strategies. And that it's weird that when using --dist you also need the --tx opción. - CMCDragonkai

IIRC It impacts the count of processes on the workers for tests execution, the --dist identify strategy for splitting test suit and sending test cases for execution to workers and tx provides way to access the workers, so there are no much sense to apply any distribution strategy if no workers specified, (actually it's debatable if we will go into details of fixtures). That's my tractation of the intend that args designed that way :) - Andriy Ivaneyko

Pero -n creates workers doesn't it? I suspect that it leads to a multiprocessing pool. - CMCDragonkai

yeah, seems controversial ( with --tx you can provided processing to me as well, i think deep dive into codebase can answer that question. I believe worker is dedicated machine whereas -n is just way to parallel tests locally and not considered to be worker - Andriy Ivaneyko

With pytest-xdist there currently no kind of "per-file" or "per-test-suite" distribution. Actually, if a per-file distribution (e.g. tests in a file will be only executed by at most one worker at a time) would already help your use case i encourage you to file a feature issue with the pytest issue tracker at https://bitbucket.org/hpk42/pytest/issues?status=new&status=open and link back to your good explanation here.

cheers,holger

Respondido el 09 de enero de 11 a las 16:01

For anyone viewing this post and wondering if any issues have been opened, the issues are: github.com/pytest-dev/pytest/issues/175 y github.com/pytest-dev/pytest/issues/738 - Chris Clark

pytest_mproc (https://nak.github.io/pytest_mproc_docs/index.html) provides a "group" decorator that allows you to group tests together for execution serially. It also has faster startup time when working on large numbers of cores than xdist and provides a "global" scope test fixture.

Respondido 25 Abr '20, 05:04

Having test suites in directory like in the question, you can run them in parallel it via:

pytest -n=$(ls **/test*py | wc -l) --dist=loadfile

If you have your tests suite files in single directory then just

pytest -n=$(ls test*py | wc -l) --dist=loadfile

In case new suite file occurs, this will include new test file automatically and will add additional worker for it

respondido 03 mar '20, 13:03

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.