The primary goal of software testing is the delivery of a bug-free product, ensuring that it meets the predefined requirements. Automation testing helps a lot in achieving this goal because automated testing reduces regression testing time across with adopting of CI\CD pipeline and allows the delivery more rapidly. Informative reports are the decisive component of this tandem.
Test reports and analytics can give the feedback that your app is of an acceptable level of quality. In other words, informative bug report is able to bring stackeholders a clear understanding that the aplication and its features are on track for release.
Hence, every time manual or automated test scenarios run during the Software Development Life Cycle (SDLC), the QA team strives to generate a test execution report of their test efforts.
Test Management System testomat.io provides comprehensive test result reporting and rich analytics with details, presenting data in an easy-to-understand form to both the business side (Product, Project Manager, Business or QA Analysts) as well as Developers.
It fosters transparent and effective communication within the entire team, whether you are working on a classical project or a Behavior-Driven Development (BDD) project. Let us remind our test management tool stands out as one of the few testing tools that fully supports BDD testing frameworks and BDD test implementation based on the Feature file as an entry project point. So, we offer QA managers and QA Leads with our testing tool:
Сheck results of the following types of testing:
- Unit testing
- Component testing
- API testing
- Integration testing
- End2End testing
- Single Result of Sync Manual and Automated Tests
By test methods successfully organise:
- Functional testing
- Smoke testing
- Regression testing
- Exploratory testing
Test reporter vs test runner and test report
That’s the description of test report implementation. Our Reporter is a specialized test reporter designed as SaaS platform and on-premise test management solution with simple console HTML kind of report in headless mode as well. Let’s deep into the next entities in which way they work together!
- Test importer – at first, you need to import automated tests into a project. Test importer connects with the folder of your automation testing framework and makes the instant import of all tests to your project in the test management system.
- Reporter – is based on the open-source library. Reporter is used to output test run reports in a certain format. Before running autotests, you should select a reporter from the dropdown list of automated testing frameworks and the type of your project, meaning classical project or BDD project.
- Test runner – is something that allows us to run the tests which are contained in your test automation framework. Test runner calls automated tests or groups of tests from the source code and then executes them and writes the test results to the console.
It supports cross-browsing, cross-platform and parallel testing - Test report link is a result of the output of your tests. Rich HTML report, custom report, thirty-part report you can name it with such terms as well.
Test importer, test reporter and runner start from the command line (Command Line Based Tool) with a few simple commands. We implemented a convenient UI/UX interface with CLI command tips on each step for each of the testing frameworks. Just copy and paste it into your console without any special settings and get a response. Also, you are able to execute tests and receive real-time test reports with Continuous Integration services, such as GitLab, GitHub actions, Jenkins, Bamboo, CircleCI and many more. And seamlessly integrate test importer, test reporter through API in your custom dashboard.
We understand that different teams and projects might rely on different testing frameworks therefore, we are constantly working on expanding multiple programming languages and integrations with new tools so that you can choose the test automation framework that best suits your needs while still benefiting from our robust reporting capabilities.
Test reporter supports programming languages:
- JavaScript
- TypeScript
- Java
- Python
- PHP
- Ruby
Test automation frameworks are available:
- CodeceptJS
- Codeception
- Cypress.io
- Playwright
- Webdriver.io
- Jest
- TestCafe
- Postman through its Newman client
- Detox a toolkit for mobile testing
- Mocha
- Chai
- Protractor
- Jasmine
As you know, these reporters can be specified via configuration testing framework on the command line. Learn how to install and configure our reporter for your test project within our test management solution, see on Docs.
What to do if your test automation framework is not supported yet?
*Alternatively, through JUnit integration XML file you are able to generate reports for any language or other testing frameworks. JUnit XML file includes test steps, links to attachments, and details of fails, passes, etc. Meet all our integrations by the following link
How to create a test report?
Full reporting capabilities is available after registration in the test management system.
Components & structure of test report
The overview report page is the HTML page that depicts results within several default widgets. Widgets represent the state of the results of testing web applications with:
- Pie charts
- Graphs
- Suites and test cases tab
- Heatmaps
- Progress bar
- Test run history logs
- Analytics notes
Metrics of manual and automated testing
Below are some of the commonly used metrics that each QA must always look for in any Automation Test Report:
- Total number of test cases
- Overall test result (Pass and Fail test percentage)
- Individual test result (Passed/Failed/Skipped test status of every test case)
- Error messages
- List of all test cases
- Test environment
- Build name
- Defects
Visual widgets are backed by analytics, featuring a series of unique testing metrics that provide a clear overview of your testing process, making it easier to identify issues, trends and levers to improve in the test framework code script.
- Test Automation Coverage
- Ever-failing tests
- Flaky tests
- Slowest tests
- Newer run tests
- Tag statistic
- Labels
Capturing Stack Trace and Exceptions in Failed Tests
In addition, it should be noted the ability of our reporter to automatically generate warning messages for the software testing frameworks it supports. Stack Trace and exception messages in failed tests indicate the place where the test script logic broke in the test code during testing.
Stack Trace and Exception messages include detailed information about errors and exceptions with red colour in the extract of the particular test automation script. This feature is helpful for test automation engineers to pinpoint the root causes of issues and troubleshoot them fast.
Custom Reports with the Specific Test Data You Require
Test Management System (TMS) empowers you with the capability to select multiple test scripts or test suites and combine them into logic groups. A sorting and filtering feature, custom labels and tags are suitable for this purpose. Thus, you can easily exclude unnecessary test cases from the test run, and then merge the results of specific test runs into a single, unified report that perfectly fits your needs. Thanks to labels, tags and filtering capabilities you can see some trends in retrospect and identify the dependencies within the History of Runs and Reports.
Advantages of Testomatio Test Reporter Compared to Other Automation Tools
- Test reporter is really fast. You can easily import and execute thousands of tests.
- Supports parallel parallel test execution and multi-environment running.
- Clean and easy-to-understand UX/UI with autogenerated tips.
- Provides synchronization with manual tests.
- Always relevant reports of test results and progress throughout the testing process.
- Archive of test executions and test reports history
- Comprehensive advanced analytics in the Test report and Analytics Dashboard.
- Allows organizing CI\CD pipelines for Agile teams.
Let’s mention a few related first-class test reporting features:
- Real-time report – test reporter sends information immediately. Possible access to test report results once even 1 test in your test framework is executed!
- Public report – instant sharing of test results to stakeholders. Public report won’t contain security-sensitive information.
- Read-only access for some team members – in addition, the system provides an option for easy sharing of reports, ensuring transparency and at the same time sensitive information remains secure.
- Analytics Dashboard – gives a possibility to track such testing metrics as Flaky tests, Slow Tests, Automation Coverage, Defect Coverage, Tag statistics etc. in the appropriate measures of the overall state of the test automation framework..
- Aggregated analytics across different test management projects – analytics dashboard collects data of all your testing projects within a company.
- Flaky test management – easily detected flaky tests of your test automation frameworks! And with a flexible “flakiness” configuration decide which tests should be flaky yourself.
- Unlimited artifacts storage S3 private cloud – test artifacts are uploaded to S3 Object Storage bucket on AWS, DigitalOcean, Azure, or Google Cloud. So, by using external storage you get full control over artifacts.
- Re-run failed tests directly from the test report – it is convenient for you and saves time.
- Link to requirements and issues directly from the test report in popular task management systems like Jira, Linear, AzureDevOps.
- Jira integration – advanced Jira plugin provides end-to-end functionality to link tests to requirements user stories. Non-technical teammates and QA engineers can view, edit, run scenarios also reproduce bugs directly from Jira. Meet all test management features
