Automated defect triaging

Effective testing is an essential phase in the software development process. To improve it, companies have begun to shift to extreme test automation. The integration/continuous delivery (CI/CD) pipeline enables higher automation coverage, maximizing the efficiency of the entire process. However, challenges persist. The Software Development Engineer in Test (SDET) often needs help with sifting through automation reports and logs and with debugging the test results. Given advancements in technology, engineers can now get help via a new automated defect triaging solution. This solution/approach is part of a concept known as test life cycle automation, which substantially reduces human error. In addition, it helps enterprises deliver bug-free applications to their users.

This article is the final piece in a three-part series about this topic. The first article, “Self-healing test automation – a step toward test lifecycle automation,” was published in July 2021, and the second, “Automated impact-based test selection and prioritization,” was published in March 2022.

Even when tests become automated, ancillary activities (like script maintenance, impact analysis, and defect triaging) can be left behind. The persisting manual nature of these activities can slow down software development. A need persists for automation that is carried out throughout the testing life cycle and that speeds up the software development life cycle (SDLC). Fortunately, the rise of artificial intelligence/machine learning (AI/ML) techniques and programming capabilities has created a paradigm shift to encompass test lifecycle automation and not just stick to test automation. Now, automation engineers can focus solely on creating new test scripts.

The need for automated defect triaging (ADT)

Within the world of test life cycle automation, automated defect triaging (ADT) is attracting interest, and clients are taking notice. Most automation engineers and SDETs follow a three-part process:

1. Identify the test scripts that failed.
2. Manually analyze test script failure.
3. Categorize issues according to their real reasons for failure.

This approach is reasonable when test volumetrics are low or when test execution is infrequent. However, it is easy to imagine a scenario in which many automated test scripts need to be executed daily. The automation engineers may become overwhelmed and need an automated mechanism that can aid in defect triaging and logging.

How does automated defect triaging (ADT) work?

Figure 1: Workflow of an ADT

The ADT solution uses an AI/ML model to auto-categorize failed test scripts into predefined categories based on test execution and failure patterns. It helps identify failed execution root causes and suggests recommended actions based on knowledge from previous executions.

It consists of the following steps:

Step 1: Test script execution result collation

Irrespective of the test automation framework used — whether based on open-source tools like selenium/extent reporter or on commercial off-the-shelf (COTS) tools like UFT — the test results need to be gathered in a commonly accessible location for the ADT agent.

Step 2: Monitoring through ADT agent

The agent will continuously poll the reports and test execution result stack trace to gather all the results and move them to a database.

Step 3: Automated Defect Triaging (ADT)

The ADT solution picks up the data that the ADT agent sends to a database. After initial data preparation, the data is fed into a classifier model, which helps determine the reason for failure. The test failure is classified based on the result description and by pattern-matching with the historical data. This classification determines if the failure was due to a configuration failure, test infrastructure failure, or a product failure. If at any moment the classifier model cannot auto-classify the defect due to low confidence, the human in the loop (HITL) comes into the picture to assist with the workflow.

Step 4: Defect de-duplication

Once the cause of failure is determined, it passes on to the defect de-duplication, and the model tries to identify similar predicate defects that already exist within the defect management tool. If the classifier model cannot succeed because of low confidence, the human in the loop comes into the picture to assist with the workflow.

Step 5: Defect logging

Defect logging is the final step in the process, closing the loop. The solution should update the existing defect if a duplicate is found, or it should log a new defect within the defect management tool.

Key benefits:

The output of the ADT solution is extensive, as it improves the following processes:

• Classification of test failures
• Root cause analysis of the defects 
• Recommendation of resolution steps 
• Identification of duplicate defects 
• Creation of a summary of test defects, including drill-down details 
• The provision of alerts/notifications to subject matter experts (SMEs) regarding critical, high-severity defects; this enables quick, decisive action 

The right-fit, popular-classification algorithms can be used for various classifier models. Moreover, a proof of concept (PoC) should be performed on existing data in order to determine the right algorithm. The ADT team should work with the quality assurance (QA) team to retrain the model regularly and triage the defects. The ADT solution helps reduce the test script failure triage efforts by a considerable margin. In a sprint model with a high level of DevOps automation, there is constant pressure on the test engineers to perform continuous testing within the sprint, and to do so effectively. Bottlenecks within the manual defect triaging process consume SMEs’ bandwidth — often when they should be focused on testing. This effect is magnified when there are thousands of automated scripts. It would be wise for companies to invest in a solution that addresses these challenges.