Viewpoint News, November 2010

Testing the Test System

by James Campbell - jac@viewpointusa.com

Those IT and web development programmers have it easy. Pure software applications are easier to test than a Test System. OK, I might be exaggerating, but the hardware involved in a Test System complicates the usual techniques utilized in software validation. When we deliver a Test System, we have to validate both the software AND the hardware.

In months past, we’ve shown you how to check the accuracy of the Test System hardware. Now, you need to verify that all the measurements are being collected, analyzed, and reported properly in the right sequence at the right time. This verification involves the application software, the unit under test (UUT), and all the measurement hardware as well.

Here are 6 tips to help you deliver a properly functioning Test System.

Lessons from the Production Floor

The environment in which IT developers work has little of the intricate hardware that product test developers encounter every time a new or upgraded Test System is deployed.

Product test developers usually can’t verify that the Test System works without an actual product to test. In other words, the functionality of the test is so intertwined with the UUT, it is difficult (impossible?) to know that the Test System works without the UUT. And this situation happens after you’ve verified the test application or test steps as best you can without the UUT. If you proceed improperly, you’ll find yourself troubleshooting in the high pressure environment of the production floor.

Software Development Testing Methods

In the software development arena, people have spent countless hours thinking about methods to assure quality (i.e., bug free) software that supplies the requested functionality (i.e., satisfies the requirements). Fortunately, much of this body of knowledge is applicable to system development. The overall scope applies and we need to amend it to address the hardware included in test systems.

As usual, Wikipedia has a very nice overview of software testing that links to other relevant topics. Refer to http://en.wikipedia.org/wiki/Software_testing#System_testing for more information. This topic is part of the broader Software Development Process content.

Here is the list of Tips followed by discussions about each.

• Tip 1: Develop a Test Plan
• Tip 2: Test the Power, Control, and Source Signals
• Tip 3: Perform Unit Testing
• Tip 4: Perform System Testing
• Tip 5: Test for Known Failures
• Tip 6: Manage Bugs

Tip 1: Develop a Test Plan

Before starting anything, even before writing one piece of code or buying one piece of hardware, develop a plan to test the system against the requirements. Thinking about testing at the beginning forces thoughts about the testability of the system. The system design may change accordingly. For example, something as innocuous as checking that the system reacts properly to a critical pressure level can require additions to the system hardware that enable the application of high pressures for the purpose of testing. Or, knowing that the system can measure signal transitions at the highest required frequency requires that you purchase a special waveform generator (which has a 12-week lead time).

Requirements Needed

NOTE: If you have not yet gathered your requirements, do that. Note that test system development often requires compromise between the design of the test system and the requirements. Of course, the test specifications for the UUT may not be changeable, but the methods used to measure and analyze those test specs can often change.

Tip 2: Test the Power, Control, and Source Signals

This tip is obvious. But maybe the reason it is obvious is also the reason it is often overlooked.

During designing, you consider power supply wattages, harness and wiring noise reduction techniques, wiring connection methods, and so on. During testing, you need to check your expectations for performance of these designs as well as verify signal integrity itself. For example, is the power supply supplying too much current for its rated output? Or is the signal-to-noise ratio sufficient to make measurement with expected precision? Is the control scheme stable and responsive? Are all the signals collected properly: triggered, synchronous clocking, in-range and so on?

Eliminate the Obvious

A rule of thumb I’ve noticed with the test system hardware is that almost 90% of unexpected hardware behavior is due to bad connections and poor power supplies. As I said, it’s the obvious that is often overlooked.

Tip 3: Perform Unit Testing

Unit testing applies to both hardware and software. The term “unit” applies to any functional subsection of the system. Clearly, the complexity of testing a unit depends on the number of inputs and outputs as well as the range of those inputs. The input ranges should be traversed to assure all paths through the subsystem are “exercised”, the goal being to test all the “corners” of the system.

The goal of Unit Testing is to test modules as they are developed rather than integrating first and testing after. Such testing will identify problems and bugs as early as possible. I often call this effort verification testing, and it is one part of the V&V (verification and validation) testing that people (e.g., FDA regulated companies) discuss.

Check out http://iitatlns2.iit.nrc.ca/publications/nrc-47445_e.html for a comparison on the application of testing methods on resulting applications.

Hardware Unit Testing

For hardware, this approach implies testing the performance of subsystems, such as wiring, instruments, and so on, as soon as possible. Do not wait until the Test Station is completely wired. I recommend proof-of-concept testing during the design stage if any concerns linger about the ability to meet measurement or control specifications and requirements.

Software Unit Testing

For software, this approach implies testing the performance of functions (e.g., VIs), frameworks, major modules, and so on as soon as possible.

LabVIEW has a Unit Test Framework tool that enables you to write and manage unit tests. This tool provides the ability to feed a test vector (list of inputs) and check the result vector (list of outputs) against the expected vector. When a mismatch occurs, the test code fails and the tool alerts the developer. The tool also provides the ability to execute a “setup” VI before and “teardown” VI after you run the unit test, allowing you to connect to hardware, or not, to create the test vector as part of your unit test. For details, see: http://sine.ni.com/nips/cds/view/p/lang/en/nid/209043.

Integration Unit Testing

When the integration of SW and HW begins, the same Unit Test approach of testing subsystems is extremely valid.

A good recommendation for integration unit testing is to use known signal sources and basic control schemes or commands to validate the connectivity between the HW and SW. This basic testing provides the foundation to trust your application before connecting a UUT. Why take this approach? Because it happens more frequently than any cares to admit that the UUT does not behave as expected under the test conditions and you want to be assured that the Test System is not the culprit when that happens.

Test Driven Development

A growing trend with software development is to write the Unit Tests before writing any system application code. This approach is called Test Driven Development. We’ve used this development method at Viewpoint of a few projects. Two main benefits appear. First, the application code is more robust. Second, the collection of test code is automatically applied to the application code during each new build. Results can be impressive (based on anecdotal evidence):

• Less than 1/2 as many bugs as usual.
• Saves over 90% of unit testing time.

See http://en.wikipedia.org/wiki/Test_driven_development for more info.

Tip 4: Perform System Testing

Once all the subsystems of the Test System are show to function, and all these parts are integrated into the completed Test System, it is time to test the Test System as a complete entity. I often call this effort validation testing, and it is the other part of the V&V testing.

System testing encompasses two main efforts:

• Test for coverage against all the requirements.
• Test for robustness over many UUTs and many hours of usage.

Regarding the requirements coverage, create a traceability matrix. There are several tools to assist with this task. NI has Requirements Gateway, IBM has RequisitePro which became part of IBM after they bought Rational, IBM now also has DOORS which has been a long-standing tools from Telelogic, and Siemens has TeamCenter which is an all encompassing suite for product lifecycle management (PLM).

The robustness testing focuses on assuring that all the previously tested subsystems work together as expected and they work consistently over time. The time aspect is important because hardware may work fine in the short term but be susceptible to overheating or external noise factors not seen in short runs. Similarly, everyone knows that software becomes more prone to failure after time.

Tip 5: Test for Known Failures

Clearly, testing for expected behavior is important. However, it is also important to know how the system behaves when failures occur. Handling failures leads to robust system behavior, and such error handling needs to be incorporated into the system architecture from the start. Likewise, when tests are performed on the UUT, you need to know that the unit is failed when it is not performing properly. Testing for failed performance can be as simple as verifying that the test system limits are set correctly and as complex as knowing that the analysis algorithms work with unexpected signals. The idea is that there should be no false passes.

Hardware Failures

However, testing that the failures are handled properly can be very challenging, especially when hardware and a UUT are involved. The reason is simple: injecting faults into hardware may result in destroying the hardware. For example, to show that your Test System handles loss of communication from a UUT may require that the UUT communications chip be disabled (cut traces, mangled wire harness, ???).

There are two recommendations I can make for managing hardware failures.

• Use a Hardware Abstraction Layer (HAL)
• Create a Simulator.

Include a HAL between your test application code and the hardware itself. With a HAL, you can inject faults into the hardware I/O data stream without needing to re-verify the rest of the application. Also, when the hardware I/O needs tight timing, create a hardware simulator. We’ve used this method many times with LabVIEW RT and FPGA. It works great.

Tip 6: Manage Bugs

To assure all the problems detected during testing are resolved, you need a way to track the bugs: what is (are) the condition(s) when the bug appeared? What is the state of the bug (e.g., evaluation, fixing, testing, …, resolved)? What was done to resolve the issue?

We’ve used Trac with some success. The biggest challenge to managing bugs is the time required to enter the information about the bug. It certainly takes discipline.

Remember that this management applies to both HW and SW.

Conclusion

Testing the Test System properly provides assurance that the product test is performing as required and the system acts robustly. Some regulatory environments (e.g., FDA) required this testing as part of the overall V&V of a product test. Even in unregulated situations, testing the Test System saves time and money because you assure that it works right every time.