No room for risk
For this client in the FDA-regulated medical devices industry, testing requires a meticulous approach and comprehensive domain knowledge. This particular project focused on manual testing of a communications gateway supporting infusion pumps. It involved software, hardware, firmware, and a plethora of associated protocols and configurations.
One of our clients is a global manufacturer of medical infusion pumps and related equipment. The pumps deliver fluids such as blood, nutrients, and medications to a patient’s body in a controlled manner. The pump operator programs the rate and duration of fluid delivery through a built-in software interface.
There are three main components of the pump: the pump itself, the software inside the pump that initiates infusion, and the software that collects data about pump performance to allow for continuous quality improvement. We have helped with testing in all of these areas including:
- Software components
- Desktop applications
- Web applications
- Software for medical devices themselves
Our projects have ranged in size, user demand, and protocol:
- From small, single PC applications to fully integrated enterprise application
- From single user to 8,000 concurrent users
- From standard protocol to medical standards to custom device standards
Areas of testing have included functional, configurations, security, compatibility, stress, deployment and installation, worst case, communications, connectivity and interoperability, path testing, faults and alarms, logging, and user interface. We have touched dozens of critical apps and domains.
We have worked with this client since 2011 including their acquisition by a global competitor who had its own preferred vendors and to whom we successfully proved ourselves.
Left brain, right brain, all brain
As an FDA regulated piece of equipment, each pump including associated hardware, firmware, and software must meet specific performance criteria including data gathering and reporting. In the space occupied by our client, where your work affects the health and lives of end user patients, you can’t afford to miss anything. Each area of testing must be documented according to specific protocol and include detailed evidence of test outcomes and remedies.
This client deploys our testing teams in multiple areas and depends upon the duality of our testers. Our testers are both linear and exploratory, logical and creative. Most of our testers are trained in both testing and development. If you are only meticulous, you may miss an unintended result of a software change to a seemingly unrelated module. Likewise if you are only a tester without exposure to or experience with developers, you’re not able to lend a hand in preventing problems before they start, or in fixing a problem such that it makes things easier, not harder, for the developers to create, rebuild, or maintain a project.
Also, working on multiple projects for this client means our teams understand and can consult and cross-train with each other on the pump elements, how they function together, as well as the values and business priorities driving all work. Our domain knowledge is robust and our internal training programs mean that new team members and bench support are at the ready.
As you will see, our domain knowledge and team approach came in handy for this project because the internal team that normally would have handled it, was gone.
For this project we were engaged to manually test software within a modular workstation.The project centered around testing of a gateway workstation including a docking and connectivity system for the infusion pumps. The configuration range of the gateway was designed to make it easy to adapt to any clinical information systems/hospital environment using flexible protocols and open design architecture. The workstation was designed to serve as a communications gateway among the infusion pump, the patient data management system, and the patient monitoring system, requiring access to the infusion data contained within the pump.
- Providing power to the infusion pumps
- Providing online monitoring for infusions
- Dataset deployment
- Continuous quality improvement (CQI) log collations
- Patient data management system (PDMS) interfaces
- Nurse call system integration
- Barcode facility
Part of the compliance for the hardware itself required testing of the software and this is where we came in. (A separate Integrant dev team was responsible for software architecture, design, and deployment.)
Up until recently the software would have been tested within the client’s U.K. offices. However, about a year prior, the U.K. offices were closed for business reasons. The client turned to us for test because of our domain knowledge, internal cross-training program, and trained and available bench. They shipped the pumps to our test center and shared the source code. The pumps arrived in December and we needed to have testing completed within a few months. This would be V&V (verification and validation) testing and all of it would be manual. We would look at the feature set and ensure compliance with the specifications.
Because we were working or had worked on several projects with this client, we were already ramped up on domain knowledge. We jumped in with a team of 3 testers (eventually growing to 15 and then scaling back down), reverse engineered the software, developed a testing plan and proceeded, documenting everything along the way according to strict protocol. We worked closely with the client’s technical leads and test engineers, roughly at a ratio of 2 engineers on our end to 1 engineer on theirs.
There were three main areas that needed to be supported in our testing, each with a full suite and ecosystem of its own:
- The workstation hardware itself
- An editor and transfer tool supporting infusion safety
- Continuous quality improvement (CQI) software that aggregates data for troubleshooting purposes
We followed best practice steps in developing and executing testing:
- Initial scope
- Technical project lead and team engagement
- Project plan
- Project strategy
- Design test cases
- Approve test cases
- Formal execution
- Closure activities
Working closely with the client we completed testing for all areas within two months. Due to our extensive domain knowledge and ease of ramp-up/ramp-down, this project rolled into a related, test automation project summarized below.
Next we moved to a related support project involving test automation of a server solution that would connect all parts, related data and logs. This was the next generation of the infusion platform/ecosystem that included CQI, gateway, and pumps.
The test automation focus was a transfer tool that would allow automatic download of each pump’s infusion logs. The project would create a closed cycle from infusion set-up to analysis of related usage data and logs, with no manual intervention.
The test automation was applied to the following cycle/scenario of functionality:
- User creates dataset to include drug limits and usage profiles
- User uploads dataset to various infusion pumps
- User verifies datasets are being deployed to infusion pumps
- Gateway acts as communication point for pumps to grab datasets and deploy
- Gateway collects CQI data from pumps and sends this to server which injects into the CQI database
- The user opens the CQI application to view the logs and verify the correct usage and safety limits
Plethora of protocols make for testing challenges
A major challenge in both projects was the fact that there were about 70 different versions of hardware, pumps, gateways, and configuration communicating through different protocols. We needed to be meticulous about tracking all versions and scenarios.
Likewise, when an issue was identified we needed to understand which component or integration points within the process were failing. Was it the gateway editor, the CQI, the dataset, the server, the hardware, or a combination?
We dealt with this by following a template-driven protocol for collecting and reporting each test run, including:
- Documenting tracking specific versions of firmware version, hardware, serial number, browser and software used
- Collecting data and test evidence including screenshots and logs
- Preparing final test reports
Scaling up, down, and always at the ready
Our project manager and technical lead have been working with this client for 5-6 years. Their supporting teams and bench have an equally impressive tenure with the client and with Integrant. Internal synergies and training allow us to jump in thoughtfully where needed. We scale from 10 engineers to 25 and back down depending upon how many projects are in play.
In providing medical devices that improve and save lives, this client provides many rewards for us, both personally and professionally.