Uploaded image for project: 'eZ Publish / Platform'
  1. eZ Publish / Platform
  2. EZP-29419

Investigate ways to reduce the time taken to perform automated tests.

    Details

    • Type: Story Story
    • Status: Closed
    • Priority: High High
    • Resolution: Done
    • Affects Version/s: 2.2.2
    • Fix Version/s: 2.3.3, 2.5.0-beta1, 2.4.2
    • Component/s: QA
    • Labels:
    • Sprint:
      [2.5] - Sprint 2
    • Story Points:
      1

      Description

      At this point, it takes about 2 and a half hours for all automatic tests to be performed. We need to investigate the ways to shorten this time. Possible solutions:

      1. adding tags to test scenarios and limit the amount performed with every PR
      2. introducing performing tests in many threads
      3. maybe some another way

      We need to find the best compromise between tests duration and tests scope.

      At first, in this story, we'll take care the page-builder non-PR builds (triggered every night and by merge). Then, we'll do the follow-up for other less-travis-blocking repos.

        Issue Links

          Activity

          Hide
          Maciej Tyrała added a comment - - edited

          One of possible solutions:
          ParallelRunner
          Some examples

          Show
          Maciej Tyrała added a comment - - edited One of possible solutions: ParallelRunner Some examples
          Hide
          Maciej Tyrała added a comment -

          About travis environments and multithreading - link

          Final solution - Behat Parallel Scenario

          Show
          Maciej Tyrała added a comment - About travis environments and multithreading - link Final solution - Behat Parallel Scenario
          Hide
          Maciej Tyrała added a comment - - edited

          Not as final, as I would like it to be. Here's a report about my disappointment.

          Solution is almost ready, but the results force me to doubt, that it is the best thing we can do here.

          This extension introduces 3 annotations:

          1. '@parallel-wait' - for pointing out test cases that have to wait for tests to finish
          2. '@parallel-scenario' - test cases consecutive to each other in the feature file and having this annotation shall be performed in parallel
          3. '@parallel-examples' - for test cases of type `Scenario Outline` - examples of such test case will be run in parallel

          To introduce this solution we have to mark and organize all the test cases that we have already, with these annotations.

          Test builds for different thread settings, on admin-ui, looks like this:

          (builds are failed yet, but there is so few bugs, that we can assume, that the results are authoritative)

          Table summary:

            1 thread 2 thread 3 thread 4 thread 5 thread
          time taken 36 min 28 sec 33 min 29 sec 31 min 57 sec 29 min 15 sec 33 min 7 sec
          only TC 33 min 13 sec 28 min 33 sec 27 min 36 sec 26 min 16 sec 28 min 37 sec
          mean TC duration 14,34 sec 20,81 sec 27,19 sec 31,76 sec 39,36 sec
          percentage gain
          14% 17% 21% 14%

          Sadly, profit from multithreating is significantly reduced by performance loss. I don't know yet, if it is the fault of this extension thread management, or Travis virtual machines.

          I'll try to confront that with liuggio/fastest extension.

          Show
          Maciej Tyrała added a comment - - edited Not as final, as I would like it to be. Here's a report about my disappointment. Solution is almost ready, but the results force me to doubt, that it is the best thing we can do here. This extension introduces 3 annotations: '@parallel-wait' - for pointing out test cases that have to wait for tests to finish '@parallel-scenario' - test cases consecutive to each other in the feature file and having this annotation shall be performed in parallel '@parallel-examples' - for test cases of type `Scenario Outline` - examples of such test case will be run in parallel To introduce this solution we have to mark and organize all the test cases that we have already, with these annotations. Test builds for different thread settings, on admin-ui, looks like this: https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/483338486 - 1 thread https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/483381054 - 2 threads https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/483369552 - 3 threads https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/483336460 - 4 threads https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/483401642 - 5 threads (builds are failed yet, but there is so few bugs, that we can assume, that the results are authoritative) Table summary:   1 thread 2 thread 3 thread 4 thread 5 thread time taken 36 min 28 sec 33 min 29 sec 31 min 57 sec 29 min 15 sec 33 min 7 sec only TC 33 min 13 sec 28 min 33 sec 27 min 36 sec 26 min 16 sec 28 min 37 sec mean TC duration 14,34 sec 20,81 sec 27,19 sec 31,76 sec 39,36 sec percentage gain 14% 17% 21% 14% Sadly, profit from multithreating is significantly reduced by performance loss. I don't know yet, if it is the fault of this extension thread management, or Travis virtual machines. I'll try to confront that with liuggio/fastest extension.
          Hide
          Maciej Tyrała added a comment - - edited

          Results for Liuggio/fastest:

          Not much difference in terms of execution time.The tests are executed automatically via threads quantity equal to the cores. Which gives us 2 threads on the travis machine, and:

          • time taken - 30 min 53 sec
          • only TC - 27 min 17 sec
          • percentage gain - 18%
            (no information about mean TC sadly)

          We can as well run it with a specified amount of threads, what however I can not test right now, because of Travis issue (I'm waiting for Travis support response).

          With this, Fastest has two major pros:

          • running tests with fastest takes just 3 lines - downloading and adding the extension, and run proper command (plus a small change in behat profiles) (contra all of that plus tagging each test case in the previous solution - much harder to maintain)
          • tests take only 3,5% (1 min) more time than 4 tests threads with Behat Parallel Scenario
          • if we choose this solution, there will be a small compromise in terms of Travis logs readability, however not big enough to negatively affect our decision

          For the last time, I'll try the brute force solution and run two threads via shell, and then we should decide, which way to take, cause there is no other reasonable solution I can find for us.

          Show
          Maciej Tyrała added a comment - - edited Results for Liuggio/fastest : https://travis-ci.org/ezsystems/ezplatform-admin-ui/builds/485926845 - 2 threads https://travis-ci.org/ezsystems/ezplatform-admin-ui/builds/485905404 - 2 threads Not much difference in terms of execution time.The tests are executed automatically via threads quantity equal to the cores. Which gives us 2 threads on the travis machine, and: time taken - 30 min 53 sec only TC - 27 min 17 sec percentage gain - 18% (no information about mean TC sadly) We can as well run it with a specified amount of threads, what however I can not test right now, because of Travis issue (I'm waiting for Travis support response). With this, Fastest has two major pros: running tests with fastest takes just 3 lines - downloading and adding the extension, and run proper command (plus a small change in behat profiles) (contra all of that plus tagging each test case in the previous solution - much harder to maintain) tests take only 3,5% (1 min) more time than 4 tests threads with Behat Parallel Scenario if we choose this solution, there will be a small compromise in terms of Travis logs readability, however not big enough to negatively affect our decision For the last time, I'll try the brute force solution and run two threads via shell, and then we should decide, which way to take, cause there is no other reasonable solution I can find for us.
          Hide
          Maciej Tyrała added a comment -

          Results for bash command line threading:
          https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/486381261 - 2 threads
          https://travis-ci.org/ezsystems/ezplatform-admin-ui/builds/487388730 - 3 threads

            2 threads 3 threads
          time taken 28 min 38 sec 29 min 48 sec
          only TC 24 min 45 sec 25 min 33 sec
          percentage gain 25% 23,1%

          The gain in this case is definitely the best, but has it's price:

          • logs became absolutely unreadable - we would be forced to implement some report module
          • travis don't trace output of additional threads - we would have to deal with it ourselves
          • it's possible, that we would experience some interference between threads
          • time cap with current features is still limited to this ~24 mins (~21 mins for the longest feature + ~3-4 mins for env building)
          Show
          Maciej Tyrała added a comment - Results for bash command line threading: https://travis-ci.org/ezsystems/ezplatform-admin-ui/jobs/486381261 - 2 threads https://travis-ci.org/ezsystems/ezplatform-admin-ui/builds/487388730 - 3 threads   2 threads 3 threads time taken 28 min 38 sec 29 min 48 sec only TC 24 min 45 sec 25 min 33 sec percentage gain 25% 23,1% The gain in this case is definitely the best, but has it's price: logs became absolutely unreadable - we would be forced to implement some report module travis don't trace output of additional threads - we would have to deal with it ourselves it's possible, that we would experience some interference between threads time cap with current features is still limited to this ~24 mins (~21 mins for the longest feature + ~3-4 mins for env building)
          Hide
          Maciej Tyrała added a comment -

          To summarize everything - the best solution at the moment is the Fastest extension. It's easy to implement and gives us a certain time gain. The implementation would be done within ticket EZP-30098.

          Show
          Maciej Tyrała added a comment - To summarize everything - the best solution at the moment is the Fastest extension. It's easy to implement and gives us a certain time gain. The implementation would be done within ticket EZP-30098 .

            People

            • Assignee:
              Unassigned
              Reporter:
              Maciej Tyrała
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile