The testomat.io test management app provides flaky test detection. You can track your flaky tests on Analytics Dashboard on the particular Flaky Tests tab. In addition, the latest 5 tests with navigation to dedicated Analytics pages, you can see on the test Run Reports, learn more follow on Docs link
What’re flaky tests?
Flaky tests — are tests that random return both a passing and failing status across multiple test runs. If you retry the flaky test enough times, it will pass. But you can’t be sure whether a successful test run result of a flaky test means your code is free of bugs. Thus the test which doesn`t produce a consistent result is deemed flaky.
The problem with flaky tests erodes confidence in testing processes!
The causes of flaky tests:
- an issue with the newly-written code
- dependencies on the test execution order
- not refreshing test data between test runs
- test execution in an inappropriate environment
- timing and time zone issues
- low internet speed
- some special external factors
As you see, flaky tests are not always proof of the inappropriate quality of code. Nevertheless, the QA team m a lot of time finding a reason and test retries.
Identifying flaky tests is the first step to getting them under control. Once you know the extent of the problem, you can prioritize fixing them.
How Flaky Test Detection and flaky management work
Go to the Test Analytics Dashboard, choose the Flaky Tests tab and then set your own flakiness settings: Min < Flakiness < Max. Remind, you are free to change these parameters. By default, flakiness is detected as an average value between run statuses. If failed = 0 and passed = 1.
And as a result, you can see a table for a period of all your tests that are flaky in the flakiness frame you have selected.
Flaky test automation integration allows identifying Cypress flaky tests, Cucumber flaky tests, Jasmine flaky test, Karma flaky tests and other JavaScript testing frameworks.
Also, our test management solution helps you prioritize flaky tests by providing the following information about them: suite, test, statuses and executed date. You are able to filter this table to see test cases of interest to you. Using other capabilities of the test management app easily isolates them.
