Viewpoint News, January 2011

Part 1: When Does it Make Sense to Use a Test Executive?

by James Campbell - jac@viewpointusa.com

As the ‘Part 1’ in the heading suggests, the next couple newsletters will cover selected topics about Test Executives. Recall that a Test Executive is an application that manages the execution of a sequence of test steps, thus helping to automate a test procedure.

During last year’s Business Optimization consultations with clients, Test Executives were an important component of the discussions. I thought some content of those discussions would be worth sharing. Naturally, these topics are based on our many years of experience designing and delivering product test systems.

The first topic covers the obvious question: does it even make sense to use a Test Executive? This month outlines some important reasons to consider in the decision. Next month will review a cost scenario illustrating that using a Test Executive does not always make financial sense.

Not a Panacea

When is a Test Executive (TE) relevant? And if it is relevant from a technical perspective, does it make business sense to use one? A TE brings additional expenses in purchasing and training, and sometimes customization. These costs need to be balanced against the savings associated with test code reuse and modularization, reduction in time to develop core functionality, and standardization.

Note that I’m not including the benefits of automation in this comparison, since a custom test application also automates.

Do Not Write Your Own

Please do not write your own TE. I speak from experience. Unless you have really good reasons, it is not worth the effort to develop your own and not worth the effort to maintain this codebase. Here’s why.

Before good off-the-shelf TEs were available, Viewpoint developed and maintained 3 TEs. One was the original LabVIEW-based TE from NI modified for our needs, one was a custom script-driven TE for simple tests, and one was an object-oriented and step-centric TE for tests that required flexibility and maintainability by our customers. Over the years, we abandoned all but the last, which we still maintain and call the TEC (Test Executive Core). Why only one left? Because of ever-increasing maintenance and upgrade efforts during which the feature sets of off-the-shelf TEs improved. Our TEC survived since it still has features that other off-the-shelf TEs do not, such as a built-in Sequence Editor, with step customization, and flexible results rendering and results database capability, with the ability to incorporate any output data type, from scalars to graphs to images to whatever.

In the past roughly 10 years, many companies have developed TE options, for example National Instruments, Geotest, and QualiSystems. They are all possible platforms, but of course we have a fondness for NI TestStand and our own TEC.

Why Use a TE?

Reuse

First, Reuse of common functionality is one of the main reasons for using a TE. If you deploy tests across many Test Stations or run multiple types of products through one Test Station or both, then you will benefit from having the “boilerplate” functionality that a TE offers. If you didn’t know at the start, you will soon know that you should have used a TE after you find yourself copying code or reusing the same architecture every time you deploy a test application.

Support

Second, if your company depends on one or two people to build custom test applications, then you should worry about the developer(s) departing the company. With a TE, only test steps need to be developed and maintained. Then, people outside the company can also help maintain, deploy, and develop your test applications. Also, basing your test applications on homebrew custom applications creates support difficulties. These difficulties are especially burdensome if multiple Test Stations are in use, since custom applications essentially become a product with bug fixes, maintenance release schedules, support calls and so on.

Training

Third, with a myriad of test applications (typically with no documentation), training test engineers to develop new test applications or support other apps can be a challenge. With a TE, you only need documentation and training for the TE application. A TE offers standardization and the test engineer can focus on writing the test steps to that standard.

Development Cost

Fourth, developing an entire application usually requires a more experienced senior Test Engineer or even a Software Engineer, whereas developing test steps to fit into a TE may require only a junior engineer. Thus, the overall development cost with a TE is reduced twice: personnel cost and time to develop are both less expensive.

Feature Set

Baseline

A TE is supposed to handle the execution and management of a series of test steps that are gathered into a test sequence. A TE should offer boilerplate functionality for at least the following items:

• Management and editing of test sequences
• Ability to adjust execution flow in both manual and automatic modes
• Management of test limits
• Results reporting, both on-screen and presentation form
• Logging of test results into a file format useable for statistical and batch analysis
• Support for part and operator IDs for tracking results and sequences

Some additional features that are great to have are:

Extended

• Password control over user access rights for various levels of operation
• Deployable as executable
• Hardware calibration support
• Hardware abstraction
• Customizable sequence and step editing
• Tools for managed sequence and step releases

Conclusion

A Test Executive (TE) possibly makes sense with the existence of multiple test installations, whether on multiple test systems or multiple test sequences on a single test system or some combination. Reuse and commonality are critical criteria. Next month will present a financial analysis to estimate just how many test installations might be needed to make the investment worthwhile.