Is Test Automation Right for Your Project?

When NOT to Automate

Test automation is powerful, but not always practical. Consider manual testing when:

• Requirements or UI change frequently (automation maintenance will eat your time)

• Outcomes require human intuition (e.g., “Is this UI intuitive?”)

• One-off or exploratory scenarios

⠀“Manual testing is still critical for ensuring a pleasant and intuitive UX. Machines can’t replace human empathy.” — Chrome OS Test Plan

Pro Tip:

Automate tests that are stable, critical, and repetitive. Leave subjective or highly volatile areas for manual testing.

How to introduce a Test Automation Tool

Each organization has a testing process that needs to be understood before introducing Test automation. Some companies have processes based on strict standards because they work with systems that deal with critical information (such as banks or other financial institutions). Other companies could work on projects where time for testing is minimal, where it is not convenient to spend time on automation scripts (in this case, manual testing of critical areas would be the best approach). Each company is different, so even though Test Automation can bring benefits, we need to analyze how Test Automation will affect our current test process, including the benefits and costs of introducing the tool.

Any test tool is intended to make the testing process more effective and efficient. A test tool can provide many benefits, but some risks must be considered before introducing a tool into an organization. “Most of the risks associated with the use of test tools are concerned with over-optimistic expectations of what the tool can do and lack of appreciation of the effort required to implement and obtain the benefits that the tool can bring.”

To avoid this scenario, before introducing a test automation tool, we need to define the goals that we want to achieve with test automation. Then, we assess the costs and benefits of the tools that meet our requirements and later decide if it is worth using any of the tools.

To start, define the objectives for using test automation with the interested parties, usually the software testing team and managers; it is also very important to know the budget; otherwise, we can end up selecting inappropriate tools for our company. For each requirement, we can assign a priority or ranking that will help us evaluate which tool will meet our needs. This initial evaluation results in one to three tools that meet the requirements.

Once we have done market research and have selected a set of tools that meet our needs, we need to conduct a Proof of Concept to evaluate them and determine whether they can be used in our test process.

Steps to Successfully Introduce Test Automation

1. Define your goals:

   What do you want automation to achieve? (Speed, coverage, consistency, cost savings?)

2. Analyze your process:

   Where do bottlenecks occur? Which test cases are most time-consuming or error-prone?

3. Evaluate tools:

   Compare open-source and paid solutions. Do a Proof of Concept with one to three candidates.

4. Pilot with a real project:

   First, choose a small, stable area to automate. Measure time savings, defect detection, and feedback loops.

5. Scale up (or iterate):

   Use lessons learned to expand automation thoughtfully.

Proof of Concept

During the Proof of Concept, we will evaluate the selected tools based on how they meet the requirements defined by the stakeholders. When conducting this assessment, we need to check these considerations:

• Mandatory features that are satisfied by the tool and extra features

• The environment: some tools may only work in specific environments, and using them might require changing our test environment and infrastructure

• Technical Skills:  technical skills needed to use the tool

According to Peter Williams, “Once all proofs of concept have been carried out, it may be necessary to amend the requirements as a result of what was found during the tool selection process. Any amendments should be agreed with stakeholders.”

There are three possible outcomes from the Proof of Concept:

• None of the tools meets the requirements sufficiently.

• One of the tools meets the requirement better than the others, so we can select it to be used in our projects.

• The situation is unclear; more information is needed. In this case, the requirements need to be revised.

Once we have selected a tool, we need to check the cost of implementing it. If a vendor will provide the tool, negotiate the purchase price, annual license fee, consultancy costs, training costs, and implementation costs.

If we have selected an open-source solution, we need to analyze the time it will take to introduce the tool, the maintenance time, and whether we have support available.

We should consider the cost of introducing and maintaining the test automation tool for paid and open-source solutions. “Test tools need to be thought of as long-term investments that need maintenance to provide long-term benefits.”

Figure 2 demonstrates the costs of Test Automation. “Note that there is an ongoing maintenance cost of using the tool, but that this ongoing maintenance cost needs to be less than the cost of performing testing activities without the tool if the investment is to be worthwhile.”

If we have found a tool that meets our needs, the next step is a pilot project, where we will see how the tool can affect our current test process.

Pilot Project

In the pilot project, we will define a business case and establish measurable factors to determine whether we should use the tool.

The objectives of the pilot project are:

• Discover any changes that need to be made in processes or practices

• Determine templates, naming standards, and other guidelines for using the tool

• Estimate and quantify the financial (and other) benefits of using the tool

• Learn more about the tool to determine what can and cannot be done, along with possible workarounds

Once the Pilot project is completed, stakeholders must analyze the results to determine whether the tool will be introduced.

Wrapping Up

Test automation can significantly enhance software testing efficiency, especially for repetitive tasks and Agile workflows. However, it's crucial to strategically assess its suitability for your project, carefully weighing the costs and benefits, and conducting thorough evaluations. When implemented thoughtfully, test automation becomes a powerful tool, improving quality and optimizing resource allocation.

If you’d like help automating your testing, Oshyn has years of experience in this area, building test automation for Adobe, Optimizely, and Sitecore implementations.