Why Are User Acceptance Tests Such a Hassle?

In my years of experience as a software engineer, I have worked on various projects. In most cases, I relied on automated testing. I adhered to the testing pyramid, where unit and integration tests formed the majority, requiring only a few end-to-end and user acceptance tests (UAT).

However, there were projects where I spent nearly 70% of my time on UAT due to the nature of those specific projects, which required complex manual processes and high accuracy. These UATs often required extensive collaboration and were time-consuming. This blog post highlights those challenges and explores how AI can help streamline UAT with some initial investment.

The Recommended Approach

In software engineering, a pyramid structure for testing is often recommended to ensure quality and efficiency.

• Unit tests form the foundation of the testing pyramid, making up 70-80% of the test suite. They focus on testing the smallest pieces of code, such as individual functions, in isolation from the rest of the system.
• Integration tests occupy the next level of the testing pyramid, comprising 10-20% of the test suite. They focus on verifying the interactions between different components or modules, mocking external dependencies.
• End-to-end (E2E) tests are positioned higher in the testing pyramid, making up 5-10% of the test suite. They focus on validating entire workflows, simulating real-world scenarios from start to finish.
• UAT sits at the top of the testing pyramid, consisting of several handful test cases. Stakeholders validate these tests to ensure that the system meets business requirements.

Created on Canva by the author

The Reality of Many Organizations

In the reality of many projects, UAT often becomes irreplaceable and needs to be extensive, covering a larger part of the testing pyramid than recommended, as shown in the diagram below.

Created on Canva by the author

These are often required in scenarios where the validation is more subjective than objective. Some of the scenarios where UAT is required are:

• Third-Party Integrations: Automated end-to-end tests often fail to cover third-party integrations due to limited access and support, requiring UAT. For instance, if a system integrates with an analytics tool, any changes to the system may require stakeholders to verify the results on the tool as well.
• Regulatory Compliance: In industries such as finance, healthcare, or aviation, where regulatory compliance is critical, UATs must ensure that the software meets all legal and regulatory requirements.
• Complex Business Processes: In projects involving intricate business workflows, many UATs may be necessary to cover all possible scenarios and edge cases. For example, in a healthcare management system, processes like insurance verification are complex and involve multiple manual steps. Any updates to the system may require extensive UAT to ensure the correctness of these processes.

User Acceptance Tests are Cumbersome!

UAT follows the following steps:

• Test Case Identification: This initial step involves defining test cases based on project and business requirements.
• Engineering Testing: Engineering teams conduct initial testing to verify that the system meets technical specifications and pass on the results to stakeholders
• Stakeholder Testing and Validation: Stakeholders perform the testing and validate the test cases as passed or failed.

The diagram below gives an example of the UAT testing and shows the collaboration required between engineering and stakeholder teams:

Created on Canva by the author

This process can quickly become complex when dealing with numerous test cases, engineering teams, and stakeholder groups. This complexity often results in significant manual effort in both testing and collaboration.

Automate User Acceptance Testing

Even though UATs are cumbersome, most companies do not automate them because they focus on validating business requirements and user experiences, which require subjective assessment. However, automating UAT can save testing hours and the effort to coordinate testing sessions.

The key is automating engineering tests using frameworks like PyTest, JUnit, etc. For stakeholder testing and validation, which are more subjective, artificial intelligence can be used to simulate user interactions and assess whether the software meets business needs.

Created on Canva by the author

AI can automate stakeholder testing by simulating user interactions and validating business requirements through machine learning models. These models can learn from historical user data to predict and mimic stakeholder behavior and ensure the software meets user expectations. The code snippet below shows a simplified version of how random forest classification, a machine learning algorithm, can classify data and predict user behavior.

from sklearn.ensemble import RandomForestClassifier

# Sample data: user interactions and outcomes
user_interactions = [[1, 0, 1], [0, 1, 0], [1, 1, 0], [0, 0, 1]]
result = [1, 0, 1, 0]

# Train a model to predict user outcomes
model = RandomForestClassifier()
model.fit(user_interactions, result)

# Simulate a new user interaction
new_interaction = [[1, 0, 0]]
predicted_result = model.predict(new_interaction)

While this automation requires some investment, it is particularly beneficial for organizations where UAT is time-consuming. By automating most test cases, only a few will need to be handled manually. After automation, the testing pyramid will still require some manual testing, depending on the system’s nature, but this can be significantly reduced to focus on the key use cases.

Created on Canva by the author

Conclusion

For many organizations, UAT is more than just the tip of the testing pyramid; it requires significant time and resources before software can be deployed to production. UAT is challenging to automate due to its subjective nature and reliance on expert judgment. However, AI can be leveraged to train automated tests based on stakeholder’s behaviors and expectations. This approach can automate the majority of test cases, allowing organizations to focus on manually testing only a few key scenarios.

This article is part of The New Stack’s contributor network. Have insights on the latest challenges and innovations affecting developers? We’d love to hear from you. Become a contributor and share your expertise by filling out this form or emailing Matt Burns at mattburns@thenewstack.io.

The post Why Are User Acceptance Tests Such a Hassle? appeared first on The New Stack.