Automated Measurement System – Verifying Flow Rate Performance in a Medical Device
Assessing quality and managing traceability of a medical device
Automation increases throughput by testing multiple units independently
Worldwide supplier of products for surgery
Our client wanted to perform detailed quality checks on the performance of some of their fluid dispensing products. These products dispense fluid over long periods of time (hours) so rigorous testing of each unit in production would take too long. But, being medical devices, these products need to follow ISO 28620 standards and FDA 21 CFR Part 820 regulations, so a rigorous test process is helpful to fulfill these needs. The automated test system below is part of that overall fulfilment.
The client came to Viewpoint with the following high-level desires for a system to test the product:
Support independent configuration and testing for up to 6 units.
Simplify overall test setup by copying the configuration from one unit to another.
Handle different volume amounts supported by assorted models.
Measure from each unit the weight of fluid dispensed as a function of time. (Weight was converted to volume using the fluid density.)
Compute the “instantaneous” flow rate (volume vs time) as the test progresses.
Keep track of the calibration status of the weight scales at each of the 6 positions in the test system.
Enable some measurements to be excluded at the start and end of the run for calculations of average flow rate.
Provide graphs and metrics of results to enable faster review of the data during the test.
Add ability to comment on each unit while testing is in progress, and after, and track these for compliance.
Print (PDF) a report on each unit’s results along with its identifying info, such as subcomponent lot numbers, the test datetime stamp, and the operator’s ID.
The client had an initial version of the testing application which they developed for testing their various initial design iterations. Viewpoint enhanced this existing application with the features listed above. The motivations for this enhancement were to:
Automate testing of the initial production units more thoroughly than the existing application allowed.
Enhance the user experience for easier testing.
A major aspect of the user experience was to support testing of multiple units independently from each other so that unit testing on one device could start/stop while other devices were installed in (or removed from) the tester without interrupting tests on other units.
This need was especially important since some models might take twice as long to test as other models or the setup of one unit might need additional adjustments before starting. This meant that starting and stopping all at the same time could reduce utilization of the tester, and it made sense to have independent and parallel operation.
Furthermore, with parallel testing, the tester was not constrained to having a full set of units to begin testing operation, since the tester could run with only 1 unit installed.
The other major enhancements focused on the user experience by offering real-time data of a particular unit’s testing, as well as real-time graphs to show progress. These graphs were useful because the operator could clearly see when a unit was not performing as expected and the test for that unit could be aborted without affecting testing on the other units.
Since this testing needs to follow the requirements in the ISO 28620 standard and 21 CFR Part 820, it was important for the application to be aware of the calibration status of each of the 6 weight scales, one for each position in the test system.
The main goal of this project was to augment the test operator’s ability to set up the testing of products while providing real-time visual feedback to the operator about the testing status.
Some of the benefits of this automated test system were:
The test automation provided consistency resulting from the software-enforced test process.
Test status via the visual feedback helped increase the efficiency of the operator.
Enhance the user experience for easier testing.
The application was developed in LabVIEW and measurements were made via RS-232 communications to each of the 6 weight scales. Once a unit was installed in the tester and the operator started the test on that unit, the application:
tared that unit’s scale,
started the flow,
and collected weight measurements frequently to build a curve of “instantaneous” flow versus time.
These “instantaneous” flow numbers were used to compute statistics such as maximum flow and average flow over the course of the entire test run.
Some of the primary functionality of this system includes:
Independent tests: testing a specific unit doesn’t interfere with testing of others.
Graphs of flow rate during the test execution: visualize the unit’s performance during the hours-long test time, not just at the end.
Calibration check: the next calibration date of each of the weight scales are maintained in a configuration file, so the operator can check that the test system is ready to run a test at a particular position.
Different volumes: since each unit is tested independently, the system can handle different volumes for each position in the tester.
Enhanced commenting: operators can enter comments about each unit (or all of them) and have these comments archived for compliance purposes.
Logging of weight versus time for up to 6 test positions via RS-232 communications to scales.
Manage the configuration and commenting of the test.
Save, recall, and copy configuration information to ease the operator’s setup effort and time.
Display graphically the curves of flow versus time while the test progresses.
Compute statistics on the flow after test completion.
Multiport ethernet to RS-232 serial communications.
Custom Automated Test System – Quantifying Energy and Durability Performance for Refrigeration
Automation reduces manual labor while improving traceability
Assessing performance for improved energy ratings and longevity
Client – Zero Zone – Commercial refrigeration systems manufacturer
Zero Zone wanted to improve the capabilities and durability of their new reach-in refrigeration products.
You might think that refrigeration is a mundane product line, but that is just not true! So many innovations are occurring as manufacturers are redesigning their products to improve their environmental footprint through better energy efficiency, coolants, and durability.
Assessment requires an understanding of the performance of the refrigeration units under many conditions. Zero Zone was taking measurements with a datalogger with too few channels, and no synchronization, to other devices that feed into the system. Plus, they had multiple models of their reach-in refrigerators that needed to be assessed. Furthermore, simplifying the data collection and analysis would make it easier to validate against ASHRAE standards.
Zero Zone came to Viewpoint with the following high-level desires:
Expand the measurements by adding more channels and channel types (e.g., 4-20 mA, ±10 VDC and digital I/O).
Provide graphs and KPIs to enable faster analysis of the data during the test.
Minimize the chance of data loss during long test runs.
Synchronize data collection and actuation.
Automate storage of measurements per a user-defined period to eliminate manual start/stop of data collection.
Simplify the manual configuration setup.
Enable a way to find relevant data perhaps months or years after the test run.
Viewpoint developed a monitoring and control durability test system that could exercise Zero Zone’s refrigerators through hundreds of operation cycles over multiple conditions to simulate actual usage in, for example, a grocery store.
During initial conversations, we collaborated closely with Zero Zone to brainstorm on some potential approaches. We made some suggestions that could satisfy their desires while also managing their time and cost budgets.
For example, by automatically populating the cells in an Excel template based on their original systems’ Excel spreadsheet, we provided streamlined report generation without having to rewrite all the calculation code embedded in their Excel file in another app. The compromises we jointly endorsed were:
Run an app on a PC to configure and monitor the test.
Use both NI Compact RIO and Compact DAQ to enable robust and synchronized data collection and control with the ability to expand channels by adding modules in both the cRIO and cDAQ chassis.
Store data on a local PC rather than a remote server to minimize the probability of data lost during the test run.
Save configurations into Excel files for recall, and cloning, of prior setups.
Write measurements automatically into the same Excel file for archive of the test setup and measurements.
Create, in this same Excel file through cell formulas, the summary report from the summary calculations. This approach allowed flexibility for changes to internal and external test standards.
Upload the summary data and test reference info into a SQL database for data management and long-term test statistics.
Digital outputs (DOs) were used to control various aspects of the test, such as door open/close and defrost on/off cycles. For flexibility, the user can specify the sequencing of these DO channels, in the Excel file used for the test, with various parameters that define the duty cycle, period, number of cycles, and start delay. The timing of these DO state changes was synchronized to the data acquisition by the real-time loop in the cRIO.
This system was deployed to 6 test bays, each one of which might be testing a unit for as little as a few weeks or as much as a few months.
The main goal of this project was to reduce the effort and associated human error in the design and execution of the test run.
Some of the primary benefits for this automated system were:
Reduced Errors: pre-verified template files used for test configuration and data storage lent consistency to test setup and execution.
Less Testing Time and Effort: the automatic execution of the test and storage of measurements enabled running tests for multiple days (and nights) without technician interaction. Technicians could work on setting up other units for test rather than babysit the existing test. On average, based on the duration of the test time, testing throughput increased by approximately 25% to 40%.
Shorter Reporting Time and Effort: reports were available about 85% faster than the time previously spent creating manually. The quicker feedback saved costs through early detection of unit problems and faster teardown at the end.
Some additional major benefits were:
More details on refrigerator operation: “Wow! We never saw that before.”
Database consolidation: statistical analysis takes hours not days and includes all tests run in the lab, not just ASHRAE tests. This central database enables long term retrieval of all test data.
Reuse: techs embraced ability to reuse and modify previous setups.
Consistency: driving the test definition through an Excel file encouraged uniformity.
Traceability: documented and timestamped calibration measurements.
Flexibility: channel counts, acquisition modules configuration, calibration, and calculation formulas were straightforward to change for new test setups.
The test automation provided by this system greatly reduced the labor involved in configuring, running, and analyzing the test run. Furthermore, the customer benefited from the consistency that resulted from the software-enforced process.
We developed the application in LabVIEW and LabVIEW RT combined with a cRIO connected to a cDAQ via TSN Ethernet.
The data acquisition modules slotted into the cRIO and cDAQ chassis handled the I/O to the customer sensors and actuators. The sensors mostly measured:
Data logging of between 50 and 150 channels and control via digital signals
Interface with Excel files for configuration, data logging, and summary calculations
Replacing Wire-wrap Boards with Software, FPGAs, and Custom Signal Conditioning
Electronic components of fielded systems were aging out Reverse engineering effort converted wire wrap boards to FPGA-based I/O
Client – Amentum – A supplier for Military Range System Support
Amentum (www.amentum.com) supports a decades-old system deployed in the early 1980s. While the mechanical subsystems were still functioning, the wire-wrapped discrete logic and analog circuitry was having intermittent problems.
Systems designed and built decades ago can sometimes have wonderful documentation packets. Nevertheless, we’ve been burned too often when the docs don’t incorporate the latest redlines, last-minute changes, or other updates.
The replacement system needed to be a form-fit-function replacement to land in the same mounting locations as the original equipment with the same behavior and connections. Below is an image of the existing wire-wrap boards and their enclosure. We had to fit the new equipment in this same spot.
Figure 1 – Original wire-wrap boards
Finally, Amentum wanted to work with Viewpoint in a joint development approach. While our joint capabilities looked complementary, we didn’t know at the start how well we would mesh with our technical expertise and work culture – it turns out we worked extremely well together as a team and neither one alone could have easily delivered the solution.
Since the team treated the existing documentation package with suspicion, we adopted a “trust but verify” approach. We would use the documents to give overall direction, but we would need details from the signals to verify operation.
Leveraging Amentum’s experience with the fielded systems, the team decided early on to record actual signals to understand the real I/O behavior. We used the system’s “test verification” unit to run the system through some check out procedures normally run prior to system usage. This verification unit enabled us to use a logic analyzer for the I/O to and from the discrete logic digital signals and an oscilloscope and DMM for the analog signals. The available schematics were reviewed to assure that the signals made sense.
With a trustable understanding of system operation, Amentum created a requirements document. We jointly worked on the design of the new system. There were both an “inside” system (in a control shelter) and an “outside” system (in the unit’s pedestal).
Some overall tasks were:
Viewpoint recommended an architecture for the inside application running on PXIe LabVIEW RT and FPGA layers.
Amentum created the system control software on a Linux PC.
Viewpoint developed the more intricate parts of the inside application and mentored Amentum on other parts they developed. This work recreated the existing discrete logic and analog I/O using PXIe NI FPGA boards.
Viewpoint designed custom interposer boards to connect harnesses to the NI PXIe equipment, including a test point and backplane boards.
Amentum designed and developed the cRIO-based outside system application and Viewpoint created a set of custom interposer boards to connect harnesses to the cSeries modules.
The PXIe FPGA boards handled the required 60 MHz clock-derived signals with correct phases, polarity, and so on. Furthermore, the wire-wrap boards were register-based so the PXIe had to decode “bus signals” sent over a Thunderbolt bus to emulate the programming and readouts from the various wire-wrap boards.
Figure 2 – PXIe replacement to wire-wrap boards
Amentum wanted to be able to support the LabVIEW FPGA VIs used to replace the functionality of the discrete logic. So, Viewpoint acted as mentor and code reviewer with Amentum to ramp them up on using LabVIEW FPGA effectively. Neither one of us alone would have been successful coding the applications in the allotted time. Joint knowledge and experience from both Viewpoint and Amentum were required.
Signal conditioning and harnesses needed to be reworked or replaced as well, of course, since the landing points for the wires were different in the new system. Viewpoint suggested a technique, which we’ve used frequently in past obsolescence upgrade projects, to create PCB boards that accepted existing connectors.
For the cRIO, these interposer “connection” PCBs plugged directly into the cRIO cSeries module. For the PXIe, these interposer PCBs accepted the field wiring connectors and converted them to COTS cables that connected to the PXIe modules. These interposer PCBs could have signal conditioning incorporated as needed. This approach significantly reduced the need for custom harnesses. All told, about 200 signals were passed between the PXIe and various other subsystems, and about 100 for the cRIO. This approach saved significant wiring labor and cost.
Figure 3 – cRIO with interposer boards between cSeries and field harnesses
The work to design and build the signal conditioning custom electronics was split between Viewpoint and Amentum. Viewpoint did more design than build and handed over the schematics and Gerber files to Amentum so they could manage the builds while also being able to make modifications to the boards as needed.
Amentum wanted an engineering firm that was willing to work along side them as a partner. Joint discussions about architecture and design led to a collaborative development effort where Amentum benefited from Viewpoint’s extensive expertise and guidance on LabVIEW architectural implementation and FPGA coding style.
The main outcomes were:
As a partner of the team, Viewpoint acted as staff augmentation by providing experienced engineers with technical capabilities that Amentum initially lacked.
This team approach delivered a stronger product to the end-customer more quickly than either of us could do alone.
The combination of Viewpoint’s and Amentum’s experience reduced the amount of reverse engineering needed due to the lack of firm requirements.
Reduction of electronics obsolescence by using software-centric FPGA-based functionality. Recompiled LabVIEW FPGA could target future boards models.
Increased software-based functionality simplifies future updates and modifications.
Decrease in number of parts leading to simpler maintenance.
Lower wattage consumed eliminated need for an anticipated HVAC upgrade.
Cybersecurity concerns were reduced by using Linux-based systems and FPGA coding.
Using software to emulate the old hardware was a critical success factor. Since the requirements were not 100% solid at the start of the project, some field-testing was required for final verification and validation. The flexibility of the software approach eased modifications and tweaks as development progressed. A hardware-only solution would have necessitated difficult and costly changes. For example, some of the changes occurred very near the final deployment after the system was finally connected to an actual unit in the field.
Emulate original discrete logic functions via FPGAs
Emulate original analog signal I/O
Overall system control via Linux PC
Maintain the same user experience as existed before
Modern application architecture for simpler maintenance
NI cRIO chassis with various cSeries modules
NI PXIe chassis with FPGA modules to handle all the analog and digital I/O via a combination of multifunction and digital-only cards
Custom PCBs for signal conditioning and connectivity
Automated Manufacturing Test System for Electronic Medical Devices
Using PXI and LabVIEW for modular testing of over 1,000 different models
Client – a medical device manufacturer and repair depot
Our client manufactures hospital patient pendants used to control bed frame, nurse calling, and TV functions. The company was also growing after adapting a business model of being a repair depot for older designs for their own and the pendants of other manufacturers. As such, their products are very high mix and medium volume.
The basic functions for all these pendant models are closely related, so the client wanted a means to build a single automated test system that could verify functionality for 1000s of models. And, since the products are medical devices, the testers needed to comply to 21 CFR Part 820 and Part 11.
The testers were designed to support the common measurements needed to test the circuitry of the devices as well as the complex signals required to drive TVs and entertainment systems. A test sequence editor was created which allowed the client to create as many test sequences as needed to test each specific pendant model by creating a list from pre-defined basic measurement steps configured for each specific measurement.
For example, each device had a power supply, the voltage of which needed to be tested. To test a specific model, a voltage measurement step was added to the model-specific sequence and configured with the upper and lower measurement limits for the power supply. The complete test sequence was created by adding and configuring other measurements test steps as needed. Each test step could also be configured with switch configurations to connect the measurement equipment, such as a DMM, to the proper pins on the device circuit board.
Using this configuration process, the client was able to support the testing of well over 1000 models without any programming. A separate application was developed to create these test sequences which were saved as XML and fed to the test system for selection and execution.
The test execution was managed by NI TestStand and the pre-defined common test steps were written in LabVIEW. The test sequences and test results were interfaced to the client SQL database which they used in their ERP system. This ERP system used the results produced by the test system to help manage the workflow of production, for example by assuring that all units had passed testing before being shipped. Part 11 compliance was handled through checksums used to check if results had been modified.
Test sequence editor used to develop and maintain tests for 1000s of device models
Enabling our client to create test sequences without programming reduced overall development costs by about 50%.
Test sequences and test results were stored in the client’s ERP SQL-compliant database for integration with manufacturing workflow
Modular and common software developed for the test systems reduced the V&V effort during IQ & OQ by allowing testing of the test execution application separate from the individual test sequences.
The automated test system was able to execute each test sequence in three different modes: engineering, service, and production. Each mode has been specifically designed for various departments throughout the manufacturing floor. Typically, the manufacturing engineer would verify the sequence by executing it in engineering mode. Once the test sequence parameters pass, it was then approved for production testing.
During actual product testing, an approved and digitally-signed test sequence is loaded and executed via the test sequencer, designed for automated production. During execution, test results are displayed to the operator and simultaneously pushed to a database. The automated test system produces a record for each tested device, indicating the disposition of each test step and the overall performance of the device. All result data are digitally signed and protected from tampering.
The architecture of the test system follows a typical client – server model.
All client stations communicate with a central ERP and SQL server and each computer is secured by applying operating system security. The SQL server contains all of the test definitions, device history records and results. Information from it can be queried at any time by quality engineers throughout the organization, assuming they have proper login access. This provides real time status about products ready for shipment. Also, other than the software running on the client stations, no other user has permission to write or modify any information in this database. The client is able to keep the server in a protected area separating it from the manufacturing environment while the client test stations are placed throughout the manufacturing area.
Surprisingly, there were only twelve test steps needed to uniquely configure and be combined to create sequences to test well over 2000 unique models. Test steps are capable of measuring basic resistance, current and voltage parameters as well as perform sound quality measurements and high speed digital waveform analysis. Several tests were designed to be subjective while others are fully automated and test to a specified acceptable tolerance. During configuration, each test step requires the manufacturing engineer to enter expected values and tolerance limits to define pass – fail status. Upon testing, the devices are attached to a generic interface connection box and the test system makes the appropriate connections and measurements.
Low-level measurement drivers to interface to a DMM, signal generator, switches, and data acquisition cards.
Measurement-based test steps
Test sequence execution
Test sequence management
User access management
Test report creation and management
Verification of test sequence content and ability of user to execute
Verification of the content of the test results
NI PXI chassis and controller
NI PXI acquisition cards for analog measurements
NI PXI acquisition cards for digital input and output
NI PXI DMM for precision voltage and resistance measurements
Using NI PXI and LabVIEW as a common architecture for multiple test systems testing several subassemblies
Client: a manufacturer of automated blood analysis machines
Our client was embarking on a complete redesign of their flagship automated in-vitro Class 1 blood diagnostic machine. In order to meet schedule goals, the design and build of several automated test systems needed to occur in parallel with the overall machine. In a major design paradigm shift, many components of the machine were being manufactured as modular subassemblies, every one of which was an electro-mechanical device. Thus, multiple testers were required to test each of the specific subassemblies in the machine. And, since this was a medical device, the testers needed to comply to 21 CFR Part 820 and Part 11.
With a looming deadline, the testers needed a common architecture, so that all testers could leverage the development from the others. Since each subassembly could be tested independently of the overall machine prior to final assembly, the design of the testers was based on a common measurement and reporting architecture, written in LabVIEW, that interfaced to the customers Part 11 compliant database for testing procedures and measurement results. Furthermore, procedures and validation checks for calibration of the testers were part of the overall test architecture.
Modularization of the test system architecture aided development and maintenance
Reduced overall development costs due to standardization of test sequence steps and reporting
Both test sequences and test results were stored in a managed database that satisfied 21 CFR Part 11 requirements
Modular and common software developed for the test systems reduced the V&V effort during IQ & OQ.
Since multiple subassemblies were being tested, with one part-specific test system per part, the automated test systems used as much common hardware as possible to simplify the development effort through common hardware drivers and test steps. Measurements were made with PXI equipment. Test steps and the test executive that executed the test sequence(s) were developed using LabVIEW.
The types of test steps required to verify the proper operation of each subassembly were categorized into basic operations, such as voltage reading, pulse counting, temperature reading, and communications with on-board microcontrollers. The specifics of each measurement could be configured for each of these measurement types so that each test step accommodated the needs of the specifics of each subassembly. For example, one subassembly might have needed to run the pulse counting for 2 seconds to accumulate enough pulses for accurate RPM calculation while another subassembly might have only needed 0.5 seconds to accomplish that calculation.
The configuration of a test step algorithm was accomplished via an XML description. The accumulation of these XML descriptions of each test step defined the test sequence run on that specific subassembly.
Test results were associated with these test sequences by completing the entries initially left blank in the test sequence, so that all results were explicitly bound to the test sequence.
The operator user interface distinguished between released and unreleased test sequences. With unreleased test sequences, engineers could try the most recent subassembly designs without needing to wait for final validation. The released sequences were only available to test operators. This login-driven branching was managed using the Windows login, so that the client employees could use their company badge-driven login process. Once logged in, the user would be able to execute the test sequence in automated mode, where all steps happen automatically, or manual mode, where one step could be operated at a time.
Furthermore, the Windows environment was locked down using built-in user account group policies to designate the level at which a user could access Windows or be locked into accessing only the test application.
During the V&V effort, each test sequence was verified for expected operation, against both known good and bad parts. Once verified, the sequence was validated against the requirements and, when assured to be as expected, a checksum was applied to the resulting XML test sequence file and all was saved in a Part 11 compliant database. Upon retrieval, when ready to run a test, the sequence was checked against this checksum to assure that a sequence had not been tampered.
Test results, saved as XML in the same file format as the test sequence, were also surrounded by a checksum to verify that no tampering had occurred.
The IQ/OQ efforts were handled in a traditional manner with the client developing the IQ/OQ documentation, with our assistance, and then executing these procedures, again with our assistance.
Low-level measurement drivers
Measurement-based test steps
Test sequence execution
Test sequence management
User access management
Test report creation and management
Verification of test sequence content and ability of user to execute
Verification of the content of the test results
PXI chassis and controller
PXI acquisition cards for analog measurements
PXI acquisition cards for digital input and output
Our client was already doing validation, but it was manual, and the client’s customer started requesting faster turnaround of results. Their customer was also requesting data to be sent with the results. Our client chose to automate the validation process to enhance their productivity.
Logs errors during the test (e.g., for continuous monitoring tests, logging the number of instances of when a UUT’s LIN (Local Interconnect Network) response deviates from a static, current draw outside of limits)
Capable of testing a large variety of product lines
Logs pertinent data to a database for post-test analysis/inclusion into reports
The UUT is an electro-mechanical part that falls under a variety of different product lines. As such, the client had a couple variants of the tester, based on the communication needs of the UUT. A total of more than a dozen testers were deployed. The functionality of the tester evolved over time, specifically modifying software to make the tests faster / decrease cycle time.
Extensive diagnostic/manual operation of system for debug of software and electrical connections between the UUT and the test stand/tooling.
Product-specific software components to operate unique products.
Execute mechanical endurance tests.
Execute environmental endurance tests.
Database output containing results from every test cycle (either mechanical cycles or time depending on test being run).
Multiple International Deployments Helps Prove Product Meets Spec.
Each endurance test can run upwards of 6 months.
Client: Major Automotive Component Supplier
A new endurance test system was developed to give more precision in the control setpoint. This additional precision enabled potential clients to review the product performance in real-life situations. Each endurance test can run upwards of 6 months.
The updated endurance tester supports product validation by providing the desired parameter control method, allowing the client to prove more obviously that their part met the stated specification.
Viewpoint developed the software and selected the NI hardware for the first unit. The client is now deploying copies of this system to multiple international manufacturing plants.
Able to prove meeting a particular product specification of interest
Closed loop parameter control
Emergency shutdown functionality
The cRIO-based endurance tester provides closed loop control, data collection, and alarming with controlled and emergency shutdown functions. The operator can manually configure a test or load a saved configuration. After a manual operator check to make sure the setup is operating correctly, a successful test will run its full duration and stop on its own.
Creating an N-Up Tester to handle increased production volume demands
Enhanced throughput offers ROI payback period of less than 1 year
Automotive Components Supplier / Manufacturer
The company makes automotive components in very large volume, several part models each at more than 1 million per year.
The client’s primary concern was conserving floor space. They were completely out of spare manufacturing space.
Viewpoint created an N-up NI PXI-based Manufacturing Test System. In this case, N=6 because analysis showed that a 6-up electronic part tester allowed the test operator to cover the test time with the load/unload time.
At the high volumes needed, the client needed to parallelize as much as possible. The cost of 6 sets of test equipment and device sockets was less important than speed. Using the equation:
ProfitPerUnit x NumberAdditionalPartsPerYearAfterParallelizing > CostOfTestEquipment,
being able to completely parallelize made the number of extra units per year large enough that the payback time for completely duplicating the measurement instrumentation for each UUT socket was less than about 1 year.
Paid for itself in less than 1 year by the enhanced throughput.
This approach consumed about 20% the floor space that would have been used for duplicating the test system 5 more times (for a total of 6 testers)
Viewpoint developed an NI TestStand application that ran 6 instances of the test sequence independently of each other utilizing the duplicated PXI-based test equipment. The common parts of the overall master sequence were:
Startup check for the entire test stand
Shutdown of the entire test stand
Archiving the test results into the database
Part handling was managed by a PLC and robot which delivered the parts from a tray into the UUT sockets. Digital bits were used for signaling the test sequence which parts were present in their sockets and ready to test.
Reduced test time across several products by an average of ~25% and reduced time to create paperwork by ~3x
Manufacturer of high-voltage power supplies
The client already had an existing manufacturing test system in place. They wanted Viewpoint to enhance the tester due to an increase in production volume demand. Viewpoint reviewed the existing test system and noted 3 areas for improvement:
Automation available in the measurement instruments – most of the test equipment was automatable, via some combination of serial, GPIB, or Ethernet interfaces. Furthermore, some equipment, such as an oscilloscope, had the ability to store and recall setup configurations. The test operators already used these configurations to decrease setup time for the next test step. Most test equipment did not have automated setup.
Operator time spent on each test step – the client had been through a Lean assessment and had already done a good job of timing operations. However, we specifically noted that the operator was manually connecting to the test points and manually transcribing to paper the measurement results from instrument displays.
Automating the connections – many types of product models were being tested at this test system. Connecting the test equipment to all sorts of products would require either 1) many types of test harnesses and connectors or 2) a redesign of the products to make test connections simpler and quicker.
The enhanced automated test system included automation of instrumentation interfaces, a test executive to run the test sequences, automated test report generation, and automated test data archiving for the electronic UUT.
Reduced total test time across several products by an average of ~25%.
Time to create paperwork was reduced by ~2/3 due to automated data collection.
The enhanced test system included the following updates:
Test sequence automation
Automated test report generation
Automated test data archiving
Automation of instrumentation interfaces
Configurable automated test steps associated with each type of measurement instrument. The test operators would create a sequence of steps to setup each instrument and record the resulting measurement. The sequence of steps could be saved and recalled for each product to be tested, so the instruments could be used automatically.
New programmable meter – integrated the new DMM meter with a programmable interface to replace the one that was not automatable.
Foot switch integration – Since the connections to the test points were manual, a foot switch allowed the operator to take the measurement and advance to the next step.
The StepWise test executive platform managed the multiple test procedures created for the different products. StepWise also handled creation of HTML reports for every part tested.
It did not provide ability for unattended operation
The thermal control had to be set manually
They wanted to do less manual review of the data
The client develops mission-critical products, so there’s a desire to reduce manual operations because they have to explain any anomalies, and manual operations are typically more error-prone. They needed repeatable results that they could trust.
Viewpoint developed a new test system that utilized new hardware and software, augmented by existing low level hardware and firmware. The test system was developed to perform both functional test for production and environmental testing, and was designed to handle up to 4 DUTs at once. The test system utilizes the StepWise test executive software with custom test steps, which allowed the client to create their own highly configurable test sequences. The system was developed in two phases, with the second phase adding support for a FPGA expansion backplane (NI CompactRIO chassis) in order to provide future capability for bringing some of the microcontroller sequence activity into the NI space. In addition, the previous version had a mix of serial, TTL, and USB instrumentation, which was not as robust as Ethernet based instrumentation. Phase II involved upgrading to all Ethernet based instrumentation, and did away with the original test system’s many manual toggle switches that could be used instead of the programmable mode through the SW.
~40% test time reduction per unit
~25% reduction in anomalies that needed to be justified
Our client had an old manufacturing inspection system (really two systems: one inspection system and an assembly/inspection system) that would no longer be supported by IT and was going to be removed from the network. They needed the operating system updated, so they decided to take this as an opportunity to port the old code from VB to C#.NET, as well as update some hardware.
As migration projects often do, this effort began by working with the client to solidify requirements, followed by a reverse engineering effort to understand the old system to try to make it match the new system as much as possible.
The updated manufacturing inspection system (one inspection system and an assembly/inspection system) included a new operating system, ported code, new motion control software, new machine vision software, and a new GUI.
OS Update – Updated operating system that is supported by the IT department and is less of a security risk
Software Porting – Ported software to more maintainable language
Measurement Accuracy – Increased inspection measurement accuracy for sub-set of measurements
New GUI – improved operator user experience by improving readability, reducing # of required button clicks, and adding auto scroll functionality
Report Generation – maintained existing format to interface with customer database
The device under inspection is essentially an image sensor array used for scanning images in high end commercial-grade scanning printers. The inspection system utilizes machine vision and precision motion control to verify the location & orientation of several parts, with measurement accuracy measured in microns.
Client: A major manufacturer of data-critical three-phase uninterruptable power supplies
A major manufacturer of very large three-phase uninterruptible power supplies (UPSs) needed better measurement, analysis, and report generation capabilities. Their clients used these UPSs on mission critical equipment, such as data warehouse server farms, communications equipment, and so one. Existing testing procedures used equipment that did not allow for complete simultaneous coverage of all sections of a UPS unit, from input to output. Our client wanted a better understanding of the signals on each of the three phases at various locations within the UPS, especially when power sources were switched or faults were induced.
Also, in the prior test procedure, factory acceptance reports were manually assembled for our client’s end-customers, delaying the final sign-off. Finally, since the end-customer might want to run a specially configured test or run a series of tests in a different sequence than some other end-customer, our client wanted to be able to rerun certain types of tests or run tests in a customer-specific order. Thus, the test sequencing needed to be flexible and editable, possibly on the fly.
Finally, synchronization between the data collection on all signals was critical to assess functionality, since all 3-phases of the UPS output needed to be in the proper timing relationship.
At a high-level, the majority of testing a UPS relies on knowing the reaction of the UPS to changes on the input side (such as a grid power outage) and changes on the output side (such as an immediate heavy load). Thus, many of the tests performed on a UPS deal with power quality measurements, such as defined by IEEE 519 or IEC 61000 series standards, which cover both continuous and transient operation. The StepWise test execution platform was utilized to allow the customer to develop arbitrary test sequences using the application specific test steps developed for the program.
Our solution used a cRIO to measure both current and voltage from each leg of the 3-phase power (and neutral) by using appropriate cSeries modules connected to various voltage and current test points within the UPS. The cRIO had enough slots to allow a single cRIO to measure a single UPS.
Assessment of continuous operation mainly reviewed the UPS output power quality. Here, it was important to know the amplitude and phase of each leg of the 3-phase power. Synchronous data acquisition between all voltages and current channels was needed for proper timing alignment of collected data points.
Assessment of transient operation was often a review of power ripple and recovery time. For example, in the event of grid power loss, a UPS would switch over to backup power, with the result being a small transient created on the output a UPS. Again, the voltages and currents needed to be collected synchronously to assure that event timing was aligned.
For increased power capacity, the UPSs could be connected in parallel. When ganged together, the continuous and transient behavior of each UPS needed to be compared to the others, in order to capture the behavior of the entire combined system. Consequently, each cRIO (one per UPS) had to share a clock to enable synchronous data collection across all cRIOs. A timing and synchronization module was placed into each cRIO chassis with one cRIO acting as the master clock source and the others being slaved to that clock.
The overall test system architecture has a master PC communicating with each cRIO. Each cRIO was placed in certain activity states by the master PC, such as “arm for measurement”, “transfer collected data”, and “respond with system health”. This arrangement enables the number of cRIO to shrink or grow depending on the number of UPSs being testing in parallel.
The test system connected the timing module in each cRIO in a daisy-chained configuration, leading to data sampling synchronization error of less than 100 ns between all cRIOs, which translates to about +/-0.001 degree phase error for 60 Hz power signals. This timing synchronization was more than sufficient to analyze the collected waveform data for power quality and transient structure.
LabVIEW was used to create various configurable test steps that could be executed in random order as well as in an automated sequential manner. Our client was thus able to test a UPS in a predefined manner as well as react rapidly to queries from their customer when they were viewing a factory run-off test. For example, the customer might ask to re-run the same test several times in a row to validate consistent responses.
Each type of test included automated analysis routines that numerically calculated the relevant parameters against which the UPS was being checked. Not only was this automated calculation faster, but it reduced mistakes and improved reproducibility as compared to the previous post-testing partially manual calculations.
Data from all tests, even repeated ones, on a given UPS were archived for quality control purposes and made a part of the device history for that UPS.
Finally, the report generation capability built into this test system was far superior to the previous methodology by allowing our client to hand their customer a professional report package practically immediately the testing was complete. Customer satisfaction was improved substantially with this state-of-the-art test system.
Client: A major manufacturer of implantable cardiac and neural stimulators
Our client needed several extremely reliable test systems to test the batteries that power their implantable medical devices. These new test systems were needed for two main reasons. First, the needed to upgrade existing obsolete test equipment, based on antiquated hardware and software. Second, new battery designs could not be tested on the old equipment.
A critical aspect of the new test system was the need to detect any excessive charge being extracted from the battery, thus rendering it unsuitable for surgical implantation. Thus, the test system needed to monitor the total energy withdrawn from a battery during testing to assure that it never exceeded a certain limit while also offering precise control of the type of pulses being drained from a battery.
All test results had to be stored in a database in order to maintain device history for each battery manufactured for archiving, quality control, and process improvements.
The updated manufacturing test system is PXI-based along with a custom micro-controller-based circuit board for some low-level control. Each PXI controller communicated to the microcontroller (uC) on the custom PCB via CAN. The uC controlled the current drain from the battery while monitoring actual current and voltage from the battery at over 1000 samples per second using a precision 6.5 digit PXI DMM. Additionally, each PXI chassis was used to test many hundreds of batteries. Signal connections were handled by several switch multiplexers. Overall control of all the PXI testers was managed by a host PC connected to the PXI controller.
Reduced test system cost vs complete COTS solution with combo LabVIEW RT on PXI and firmware on microcontroller-based custom circuit board
Enabled tight control of DUT operation on controller with microsecond level responsiveness while being supervised by higher-level PXI RT
Quick-reaction test abort capability
Test results stored to database for archiving, quality control, and process improvements
In a simplified view, the testing proceeded by pulsing the battery with a series of different durations and varying amperages. The exact sequence of this pulsing is unique for each DUT model. Measurements were made using a PXI filled with various NI boards such as DMMs, for accuracy, and data acquisition cards, for general purpose use.
Additionally, the pulsing amperage levels needed to be tightly controlled in order to know that the tests have been performed properly. Thus, a real-time amperage control scheme had to be implemented to maintain the level requested for the pulse. We chose to accomplish this control via an analog control circuit developed using a custom Viewpoint-developed circuit board. This board was controlled via a Microchip PIC microprocessor. The LabVIEW RT application communicated with the microcontroller to setup the pulsing sequence and coordinate the start and stop of the pulsing and the NI acquisition hardware.
This custom circuitry also reduced the overall cost of the test system by about 40%.
The engineering time to design this custom circuitry was more than offset by the reduction in material costs because more than 10 test systems were deployed, allowing the non-recurring engineering effort to be shared between many systems.
When no critical issues were detected, the waveforms acquired by the PXI system were stored and then analyzed to determine the viability of the DUT. The pass/fail disposition, the waveforms, the total energy consumed, and other test results were then passed along to a master PC that managed all these results in a database for archiving, quality control, and process improvements, each set of results being tied to the unique unit serial number.
The test systems provided reliable operation for testing the large annual production volumes of the mission-critical DUTs.
LabVIEW RT – for managing the microcontroller functions and overall data collection and safety monitoring
Microcontroller application – to provide precision pulsing of the batteries
Communicate to the host PC – to both receive pulsing instructions and configurations and to return pulse waveforms for each battery tested.
Replacing Obsolete Custom Electronics with cRIOs in High-Power Capacitor Testing
Modular Embedded cRIO Systems Shortens Development and Reduces Risk in Complex PC-based Test System
Client: A major manufacturer of electrical power generation and distribution equipment.
This project involved retrofitting a test system used to verify operation of a high-power capacitor used in electrical power distribution. This system was originally built around 1990. Critical sections of the original test system relied on custom, wire-wrapped analog and digital circuitry to process, analyze, and isolate the high-voltage and high-current signals created by the capacitor. Analog filters, rectifiers, and comparators produced pass/fail status signals. A master PC, other measurement and control equipment, the analog circuits, and a six-position carousel were integrated to create the entire automated test and control system.
For each unit under test (UUT), test specifications are obtained from a Manufacturing Execution System (MES) and cached locally. The subsystems at each carousel position are designed to run independently. This parallel capability allows greater throughput and reduced test time per capacitor unit. In addition, as different capacitor models move through the carousel stations, the test parameters and conditions must be aware of the particular model being tested.
Test results for UUT are pushed back to the MES system for record retention and data mining. The existing MES interfaces were retained exactly for the retrofit.
All capacitors require 100% testing prior to shipment, so the test system is critical for the facility operation. Two or even three shifts are common depending on production needs and the facility cannot afford any significant downtime. Thus, a challenge was to design and build a test system that worked and was very robust.
Another huge challenge was the lack of documentation on the existing system, requiring a sizable amount of reverse engineering to understand the test system operation before development on the new system could begin.
Furthermore, one of the most important challenges surrounded replacement of substantial amounts of original test equipment before the new test equipment could be installed. Thus, we absolutely had to minimize the time and risk in this upgrade changeover.
A schematic of the overall system architecture is shown in the figure. The major components of the system are:
Master PC for supervisory control and test execution management
NI cRIOs with FPGAs and Ethernet for independent yet PC-supervised operation
Station-specific FPGA code for replacing wire-wrap circuitry functionality
Integration with existing MES, safety equipment, tooling, and measurement hardware
The architecture chosen was made very modular by the capabilities offered by the cRIO. The Master PC interfaced with station-specific measurement instrumentation as needed, such as GPIB controlled equipment, and coordinated control and outcomes from the cRIOs. This additional equipment is not shown in the figure.
The Master PC coordinated all the activities including interfacing with the existing MES database and printers at the manufacturing facility. In addition, this PC provided the operator interface and, when needed, access to engineering screen on a diagnostic laptop.
The cRIOs were essential to the success of this test system. Each cRIO functioned as the equivalent of a high-speed standalone instrument.
The cRIOs at each carousel test position had to provide the following features:
Digital I/O for machine feedback, safeties, and fault conditions
State machines to coordinate with external commands and signals
Perform numeric calculations to emulate the old analog circuitry
Control loops for currents associated with voltages needed by different capacitors
Communication support with the master PC
Computation and detection of internal fault and UUT pass/fail conditions
We were able to duplicate the behavior of the wire-wrapped circuitry by converting the schematic diagrams of these circuits into FPGA code and then tweaking that code to mimicking the actual signals we measured with data acquisition equipment on the original test hardware.
The outputs of the circuitry were reconstructed on the FPGA with band-pass filtering, calibration compensation, point-to-point RMS, and phase & frequency functions. This functionality was implemented in fixed-point math and the 24-bit inputs on the A/D provided sufficient resolution and bandwidth for a faithful reproduction of the electronic circuitry. These embedded cRIOs provided a very effective solution to what otherwise might have required another set of costly and rigid custom circuits.
Finally, for optimizing the task of replacing the old equipment, we used a set of cRIOs, not shown in Figure 1, to provide Hardware-In-the-Loop (HIL) simulation of the manufacturing and measurement equipment. These cRIOs imitated the rest of the machine by providing inputs to and reacting to outputs from the embedded cRIO controllers, thus supporting comprehensive verification of the new test system before the tear-out of the existing hardware. Furthermore, these HIL cRIOs enabled fault injection for conditions that would have been difficult and possibly dangerous to create on the actual equipment.