Testing is an essential part of the development lifecycle. The goal is to provide a final product’s quality and services that inspire the customers most while maintaining velocity. Thereof the lead cornerstone of testing today are Agile, Automation, DevOps and CI\CD workflow. And, of course, “good” reporting goes along with them, as it is a primary supplement for all.
What’s test report?
Meaning test result report, test report is a mechanism of evaluating the state of the product.
Software test result reporting can be at various stages of testing like system, integration, regression, e2e, smoke testing etc. Also can serve as a recap document of the exploratory testing session. It should be informative and understandable for all teammates, furthermore, built at the right time.
Test Automation management real-time reporting support
Web-based test case management solution testomat.io provides Real-time test result reporting. This QA report allows feedback just one test is completed. QA team need not wait until all tests will be executed. It’s a perfect option for high-scaled testing projects, in case test running goes for hours. Developers can start bug fixing till the test will finish.
Thanks to the advanced test reporter and a number of integrations, Real-time reports are available for the next testing frameworks: CodeceptJS, TestCafe, Cypress.io, Cucumber, Webdriver.io, Jasmine, Protractor, JestMocha, Chai, JUnit. Supported programming languages Java, Python, JavaScript, Ruby, PHP, CSharp and .Net.
To better inspect a test artifacts can be attached to a test. Reporter will upload files to your own S3 Object Storage bucket on AWS, DigitalOcean, Azure, or Google Cloud in interoperability mode and sends links back to the test management system and it displays artifacts in tests for watching.
The next artifacts can be recorded and linked to test report:
- Screenshots
- Video recording
Test artifacts can be uploaded with public (default) or private access to a bucket.
Components & Structure Of Test Report
Reporters of the test management tool build a comprehensive real-time quality assurance report. This real-time report features a few logic sections, which contain:
- Pie chart – shows how many tests were executed, passed, failed or skipped in this certain run.
- Test run chart – curve accumulated some historical data of test run and shown shares of failed, passed and skipped tests in a timeline.
- Spot indicators – once again they represent a counting of executed tests.
- Suites and test cases tab in the tree-like data structure. Tree is collapsible for comfort.
- Test case page – you can go to the test case page after clicking on the individual tests from some of the results overview pages
- Heatmaps – visualize the distribution of passed, failed or skipped tests per suites.
- Progress bar – visualizes the progress of testing activities.
- Test Run Archive and Group Archive – here you can track your previous test executions. These history Archives are unlimited.
- Analytics and metrics – built-in quality analytics widgets allow you quickly detect anomalies in tests.
The overview report page is the HTML page that depicts results within several sections and widgets. Different tabs allow switching between the views of data structure thus giving another perspective. Some sections in the test report support filtering and searching, also through tags and labels. Test environments are marked as a tag as well. Well, let’s look briefly at a few features we are able to launch directly from the test report:
- Stacktraces and exceptions – trace errors in program code. Exception informs us about the presence of error. Stack trace, meaning Stack of Call directly points on the part of code that caused this error.
- Rerun failed tests – you can re-execute only failed automated tests in CI\CD with a click on a button.
- Run automated tests as manual – this specific feature provides the capability to execute automated tests as manual.
- Jira integration – with the advanced Jira plugin, you are free to create or attach tests to Jira issues directly from the test report by clicking the plus add button.
- API integration – allows creating custom built-in QA reports on your environment (means Dashboard).
Who a software test report is for?
- QA team
- Software development team
- QA Managers
- Business side (Business Analyst, Project Owner, Project Manager, Project Lead).
A good Quality Assurance report should allow everyone participating in the development process to extract the maximum useful information from the everyday execution of tests. The value of reporting is that a good report helps the customer to take important decisions about the release of the product. The level of the report’s detailedness depends on the needs of a particular customer. For instance, the Project Manager or Development team needs less granularity to see a high-level picture of the product at a glance. QA Manager and QA team vice-versa a more qualitative picture with various parameters.
Given above, in terms of the audience, we provide 2 different reports:
- Full test report
- Public read-only test report doesn’t contain secure sensitive info. It is an overview HTML report page that depicts results within several default widgets.
Spread report within teammates:
- Share report by email.
- Download as a spreadsheet.
- Sending QA reports with notification Slack or Microsoft teams messages.
What about test reporting for Continuous Testing?
Performing the whole project test management inside a single test management tool it’s a great idea, isn’t it?
When you’re releasing quickly, often, and with the help of test automation — as most modern organizations do — smart Real-time reporting and analytics are a necessity.
So, you can sync your automated tests with your CI\CD pipeline. Run tests and get actionable test result feedback in the test management system within supported CI\CD integration Github actions, Gitlab, Bamboo, Jenkins, CircleCI and etc tools.
Importance of test reporting and analytics across test management
- Single comprehensive test report combines automated testing and manual testing results. Ensures that any tests won’t be lost.
- QA team members receive relevant test results. Identify critical information about testing, such as the scope of testing, the tests that were performed and the tests that were failed. Track progress throughout the testing process.
- Test report is generated and updated automatically and testers may track key activities carried out during the testing lifecycle for any release and be sure of its reliability.
- Detailed result reporting brings transparency to the testing process, which the business side would appreciate. And they can make informed judgments about whether a product with clean and easy-to-understand UX/UI QA report interface.
- Advanced Analytics Dashboard with set of metrics gives a possibility to track such testing metrics as Flaky tests, Slow Tests, Automation Coverage, Defect Coverage, Tag statistic etc.
- Allows organizing CI\CD pipelines for Agile teams. Test automation real-time reporting is necessary for running effective CI\CD pipelines.
