What are Key Aspects of Risk Management in Testing?

Firstly, I would like to refer to ISTQB—the International Software Testing Qualifications Board—where a fairly accurate definition of the main terms related to Risk Management, the topic of this article provided:

→ Risk: “A factor that could result in future negative consequences”
→ Product Risk: “A risk that impacts the quality of a final product”
→ Project Risk: “A risk that impacts project success”

Why ISTQB is now focusing on Risk Management?

In my opinion, it’s because one strives to align with market trends and synchronize with different certifications and standards. Hence, the ISTQB syllabus is a solid foundation for understanding the fundamentals of risk management and its starting practical implementation within project processing, especially in terms of high-scaled projects.

To work with risks, risks should not be isolated from other standard project activities — estimation, planning, and resourcing. When we calculate the necessary resources or time, we try to foresee when we will reach the final point of the project — this prediction is important for us as QA managers and our stackeholders. At this moment, we also assess what will help us and what will hinder us. The latter is potential risks, which we will try to avoid and I will discuss further.

Probable Risks and Key Factors Characterizing Them

ISTQB provides another, more detailed definition of risks, as well as lists their characteristics:

Risk is a potential event, hazard, threat, or situation whose occurrence causes an adverse effect.

Two factors can characterize a risk:

• Risk Likelihood – the probability of the risk occurrence (greater than zero and less than one)
• Impact of the risk (harm) – the consequences of this occurrence».

Based on these characteristics, the criticality of the risk is determined, and a decision is made about what to do with it. What risk management activities exist?

Risk Management Activities

At the beginning of the material, I mentioned that many teams do not pay due attention to risk management in testing because they consider it to be complex. The reason for this opinion is that risk-based testing involves performing various tasks:

Risk analysis, which involves:

• risk identification
• risk assessment

Risk control, which includes:

• risk mitigation
• risk monitoring

The main problem of QA leaders and QA Managers I know is that they present risk as an event that already happened, not as a risk with its preventions. For instance, the team postponed product release for 2 weeks because of a problem => unexpected Event 😲

All the risks on the project should be analyzed and controlled, regardless of their type!

I will explain the types of risks further on ⬇️

What Are Quality Risks and When Do They Arise?

Product risk involves the possibility that a work product may fail to satisfy the legitimate needs of its users and/or stakeholders. Product risks might be associated with specific quality characteristics of a product, product risks are also called Quality Risks.

Exactly, when it comes to quality characteristics, we may encounter main standard problems of such Quality Risks:

• Software might not perform its intended functions according to the specification/user, customer, and/or stakeholder needs.
• A system architecture may not adequately support some non-functional requirement(s).
• A particular computation may be performed incorrectly in some circumstances.
• A loop control structure may be coded incorrectly.
• Response-time may be inadequate for a high-performance transaction processing system.
• User experience (UX) feedback might not meet product expectations.
• e.g.

Responsible personas who control the product quality have to pay attention to them.

Managing quality risks requires a clear understanding of the software requirements that exist in the project. Various Quality Models are applied for this purpose, which teams can choose depending on the characteristics of the software project.

Quality Models based on quality characteristics:

• ISO 9126 QM. This is a comprehensive quality model that defines six quality characteristics: functionality, reliability, usability, efficiency, maintainability, and portability.
• McCall’s QM. This model defines eleven quality factors, including reliability, usability, efficiency, maintainability, testability, portability, reusability, interoperability, flexibility, correctness, and completeness.
• FURPS+ QM. This model defines quality characteristics as functional, usability, reliability, performance, supportability, and design constraints.
• Boehm’s QM. This model defines eight quality characteristics: correctness, reliability, efficiency, integrity, usability, maintainability, flexibility, and testability.

However, not all models that work with the product are focused on the quality end product and are based on quality characteristics. Among such models, the following can be highlighted:

• Crosby’s QM. This model defines quality as conformance to requirements and zero defects.
• Juran’s QM. This model defines quality as fitness for use and focuses on meeting customer needs.
• Taguchi’s QM. This model focuses on reducing variability in products and processes to improve quality and customer satisfaction.

Each of these models provides a set of characteristics, helping identify specific test suites that enable us to target and handle potential issues.

Definition of Project Risks and Factors Affecting the Project

→ Project risk involves situations that, should they occur, may have a negative effect on a project’s ability to achieve its objectives.
→ Project risks may affect both development activities and test activities.

In the context of the project, it is one service or another that will be delivered to the customer.

Let’s talk about what can affect the project more detail:

• Project issues. Late changes may result in substantial re-work. These can occur when the testing team reports critical, blocking defects too late. To avoid this, it’s necessary to ask oneself: What could have been done to detect this defect earlier? In other words, working with risks and effective planning help avoid late changes.
• Organizational and Political issues. Personnel issues may cause conflict and problems. Almost every project faces situations where testers disagree with the development team or managers. This leads to a QA team not receiving timely information, and the emerging problems are not addressed in the early stages of the software development life cycle.
• Technical issues. The test environment may not be ready on time. Often, alongside risks, many teams also ignore the test design phase. However, this phase is crucial because it involves preparing an independent test environment. In turn, the quality of the environment determines whether the team can pass all quality gates and achieve the expected quality final product.
• Supplier issues. A third party may fail to deliver a necessary product or service, or go bankrupt. Sometimes, developers use third-party components to add or change the project. Later, it may turn out that this component is no longer supported. This is a very common problem that is often not considered in the risk management plan.

All these issues can hinder the timely release of your product. Also, all project risks can be categorized into 4 categories:

In this material, we won’t delve into each category in detail because our goal today is to familiarize ourselves with the overall concept of risks and begin working with them.

Effective Risk Management

🤔 How to Mitigate Risks and Reduce Risk Probability?

According to the ISO 31000 standard, managing risks involves various approaches to dealing with them:

→ Avoiding the risk by deciding not to start or continue the activity that gives the risk.
→ Accepting the risk to pursue an opportunity – Removing the risk source.
→ Removing the risk source.
→ Changing the likelihood.
→ Changing the consequences.
→ Sharing the risk with another party or parties (including contracts and risk financing).
→ Retaining the risk by informed decision.

This standard emphasizes a crucial point: all stakeholders involved should participate in risk analysis and management. Risk should not be referred only to a team responsible for designing, developing, managing, or testing the product. All parties must have an interest in this process. At least they should be informed.

A crucial role in the effective risk management process is played by detailed risk analysis. Its results let you evaluate the thoroughness and scope of testing activities.

According to ISTQB standards, this is an approach Risk-based approach. It helps the testers understand in which area testing should be expanded to prevent risk probabilities. Another one is the Analyzing Product Risk approach.

Organizing Work with Risks in QA Team

To start managing risks as simply as possible, a testing team can gather at the start of a project or a new sprint and organize a risk session. During this meeting, it’s essential to work on, thus Risk-based testing tasks include:

Risk identification

To do this, create testing checklists describing the anticipated testing areas. It’s also crucial to share past experience with similar systems or projects.

Risk assessment

Evaluate the frequency of use of the affected feature, the lack of reasonable workarounds, and the visibility of the feature.

Risk mitigation

At this stage, reduce product risk by designing effective test cases and by participating in reviews of software work products. Also, re-evaluate known risks based on additional information.

Due to potential changes during project work, such risk sessions should be repeated periodically—at least once every few sprints.

Another practice used by many QA teams to manage risks is creating a risk register, for instance with Jira. To make it truly beneficial, the risks contained must be regularly updated and compiled according to a specific template.

🔴 Please note: Risk registers are often ineffective because teams write them for specific tasks but attempt to use them across different projects. However, you should understand that risks vary from project to project, and most importantly, their impact and probability change.

QA teams that find risk registers ineffective employ a different approach to systematizing risks. It involves creating a risk matrix, which allows visualizing the likelihood and impact of risks.

A risk matrix looks like this:

To build a risk matrix, you can use Google Sheets. In this case, the matrix will take the form of a regular table where each risk is assigned a coefficient based on its likelihood and impact:

Also, for this purpose, teams often turn to specialized paid tools, such as Riskonnect. It allows the creation of a detailed table with information about all the risks in the project with a notification option:

According to the risk matrix, all identified risks can be:

• Low Probability and High Impact. This means that the event is unlikely to occur, but if it does, it will have a significant impact on the project.
• Low Probability and Low Impact. The event is minor, and the likelihood of its occurrence is low. You can simply observe such a risk during the project’s progress.
• High Probability and High Impact. The event is almost certain to occur, and its impact on the project will resemble a natural disaster.
• High Probability and Low Impact. The likelihood of the event occurring is high, but its impact on the outcome of the work is minimal.

If you don’t have the time or desire to create a risk matrix or register, there is another way to work with risks—a Risk/Issues Register Template.

This table contains specific references (such as Jira tasks) and a list of risks with categories, statuses, and the degree of probability and impact.

Based on these indicators, risk assessment is carried out. Risks are assigned a score from 5 to 25, and each cell is colored accordingly. The last column contains a response—the action we intend to take according to the risk mitigation plan.

You can use such a table during a risk session. To do this, before the meeting, it is necessary to discuss with the team the tasks you plan to work on. You can estimate the probability using the following sheme:

This method of working with risks not only allows you to manage risks within the scope of a single project but also to track historical data: analyze which risks occur most frequently, why this happens, and utilize the obtained data in strategic planning.

After applying such a straightforward method of handling risks in a project, you can consider implementing a risk-based testing approach.

Risk-Based Testing: What to Pay Attention To

Risk-based testing is an approach to software testing where the priority of executing test cases is determined by considering the likelihood and impact of potential failures. This approach allows for the detection of the most important and critical errors at the early stages of software development.

Below is a Mind Map that will help you identify risks, assess them, and create an effective testing schedule based on this data. Also, it good for ideas you and your QA team.

Summarizing

Risk management in testing is an important aspect of the software development life cycle. Effective risk management brings undeniable benefits, allowing for the detection of system failures at early development stages and thereby strictly adhering to project goals and objectives.

I hope this material will help you convince your team of the importance of risk analysis in software testing project and implement a risk-based approach in the testing process.