One way to win stakeholder support for a test automation program

When looking at introducing test automation a common question our clients face is where to start. Internal buy-in may require a “starter” project yielding quantifiable results that support further investment. 

This case involves a long-term client in the medical device manufacturing industry. Over the years we have done a lot of testing for this client, all of it manual. The testing environment is highly complex and requires high compliance to strict protocol, which usually means significant time and resources. This project commenced when the client’s development manager decided it was time to introduce automated testing to save time and resources while enhancing quality.  

But to do so she would need to show a quantifiable benefit among non-technical stakeholders from multiple cost centers. She was confident if she could demonstrate time savings she could win buy-in for more comprehensive test automation.  

Choosing a first target 

With this in mind, she and Integrant discussed potential targets for automation. Smoke testing stood out as a strong candidate. The smoke testing process for continuous deployment was manual and time consuming, and had the potential to show a good ROI quickly. 

The existing process looked like this: Once the client’s internal team had completed the software configuration management build, they had no automated way to install the application and perform smoke testing. The software configuration manager would generate the build on the production network. The user would then take a zip file for the application, place it on a virtual machine, extract the file, install the prerequisites, manually install the application, and then go through smoke testing. The process required around 2 hours.

Automating smoke testing would make continuous deployment for the application more effective and would save hours of time. Supporters and skeptics alike would see tangible, affordable, immediate benefits.

Quantifiable and qualifiable results 

And they did! Project details follow below, but the bottom line is that the time savings that resulted from smoke test automation warranted the investment.

The time required for daily smoke testing dropped from 2 hours to 30 minutes. This equated to 7.5 hours or 1 full working day of time savings per week. Over the course of a year the client saved 9-10 working weeks or 360-400 hours. For a team of 5 engineers, the client saved the equivalent of 1 entire full-time engineer just by automating the smoke test. 

By enabling automated smoke testing, the client not only saved the time it took to manually conduct the tests, automation made it really easy to run the tests frequently. This meant developers would now get continuous feedback on whether the new features they were implementing were impacting or breaking existing functionality. This resulted in additional time savings related to regression testing and rework. 

The approach we took to get the results 

Once we had identified smoke testing as the test automation target, we looked at the tests to be automated, created a proof of concept, and defined what the test automation process would look like. Having agreed with the client on these areas, we estimated the team resources and timeline necessary to get the job done.  

The team included a development team lead on the client side and on the Integrant side, a technical project lead, and two software quality engineers.  The automation process, including installation, configuration, and running the tests required 5 months.  

Our solution architecture eliminated all but one manual component. In the resulting architecture the build was generated on the production network. The network included software configuration management and CruiseControl (an open source tool setup specifically to perform continuous integration software builds). The non-production network included three virtual machines.  

  • One virtual machine hosted Jenkins, the continuous integration server used to orchestrate the entire solution 
  • Another virtual machine acted as the application server machine 
  • A third virtual machine acted as the client machine connecting to the web application  

The one manual trigger in the solution was that the user needed to drop zip files containing prerequisites and the application into a shared folder on the non-production network. Once the user dropped the zip files the automation solution was triggered.  

From this manual trigger we needed a code structure and technology that would support complete automation. To make that happen we used the following tools and technologies: 

  • Python language along with PyCharm integrated development environment for code analysis 
  • Jenkins server for continuous integration automation 
  • Selenium WebDriver for writing automated tests (mimics behavior of a real user) 
  • Robot test automation framework for acceptance testing and acceptance test-driven development 
  • White test automation framework for UI 

The automation covered all of the following features: 

  • PCI installation  
  • SQL server installation  
  • Create local domain controller  
  • Post domain configurations
  • SQL and domain logins
  • Application installation
  • Smoke testing  
  • Client VM: Windows 10 VM for smoke testing  
  • Cross browser testing: Support for IE11 and MS Edge browsers  
  • Dynamic test data: Loading data for data set management and wireless packages from a network shared location  
  • Creation of a service account to run the entire solution  
  • Log files enhancement to include metadata in robot log files  

When the solution was demonstrated among the stakeholders, the response was both enthusiastic and collaborative. What else could we improve? Where else could we automate? 

Path forward has been paved 

With this solution and the associated time savings, the client’s development manager was able to get the buy-in she needed to continue moving forward. We are now jointly looking at additional areas of potential automation. We are focused on areas where it is clear the time required to automate the test case is a small investment relative to the time it takes the team to run the same test manually.  

Knowing that the client’s original goal was to make a case for further test automation, we designed the test framework to be reusable. 30-40% of the base components we built to run automated smoke tests can be ported to any web-based application. Also, the automated deployment and installation can be re-used for any application that leverages the InstallShield application packaging and deployment tool. 

From baby step to bigger picture 

Along with completion of the project at hand, the client and Integrant jointly defined next steps: 

  • We will explore options with the configuration management team to link the production and non-production networks to automate the final and remaining manual step. We will look at options like whether the software configuration manager could drop the build from the production to the non-production network, or if we could use a VPN tunnel connection.
  • For this initial project we focused on smoke testing for team check-ins. For future projects we will enable each developer to conduct their own automated smoke test and check in their code daily.  
  • Overall, we will explore automation of other areas of testing including at the UI, database, and CQI levels. 

Working with this client within their environment, we were able to quickly show progress, win over program heads, and in the end, generate an impressive return on investment with a scalable and reusable test automation effort.