Viewpoint News, March 2011

Part 3: Considerations for Choosing a Test Executive

by James Campbell - jac@viewpointusa.com

We think TestStand is a wonderful Test Executive (TE) and we use it often, and that’s not just because we are a Select Integrator partner of NI. We even have a Certified TestStand Developer on staff to prove it. You might also remember from Part 1 of this series that Viewpoint still maintains our own in-house-developed Test Executive, the VSI TEC, and we use it often as well. I thought you might be interested in the reasons why we maintain this TE when TestStand is available.

Distinguishing Between Test Executives

Remember from Part 1 of this series that a Test Executive (TE) is useful when there will be a good amount of commonality and reuse among multiple test systems. However, differences between the feature sets of various TEs do exist and one TE can be more appropriate than another in some cases. While there are many off-the-shelf TEs and many custom in-house versions, there is one broad attribute that I think defines two basic approaches to TEs: step-centric or sequence-centric. The comparison between NI TestStand and VSI TEC is representative of this attribute.

Centricity

Both TestStand and VSI TEC combine test steps into sequences which are then executed by the TE. The centricity affects the development environment more than the execution. Specifically, is more development effort spent in the sequencer or the steps?

Sequence-Centric

If the development consumes more effort in the sequencer, then it’s likely that:

• The TE lacks features you require in sequence execution and step development support.
• The sequence execution is complex, with branching, looping, and subsequences, parallel multi-sequence processing, and so on.
• The TE development environment is rich and capable, allowing you to consider the development of customized features you might otherwise ignore.
• The steps are simple to develop, compact in scope, or highly reusable.

Do you require such a TE?

Step-Centric

If the test steps consume more development time, then it likely that:

• The TE has all the features you need in sequence execution and step development support.
• Sequence execution is simple, needing only a top-to-bottom execution of a list of steps.
• The TE development environment is focused (maybe entirely) on test step selection and configuration.
• The steps tend to be a combination of simple and complex, where the complex ones handle the looping and branching that might otherwise be handled in the TE sequence but can’t easily because of a need to access many parameter or variables, have tight HIL control loops, or need to coordinate measurements on multiple instruments at once.

Do you require such a TE?

TestStand

TestStand (TS) was developed without any specific test steps or test applications in mind. Consequently, TS is necessarily sequence-centric. In fact, TS startup, execution, and reporting are managed by sequences that are pre-written by NI. Even the process models in TS are sequences. This approach makes TS extremely flexible and, thus, somewhat complex to customize. (Our Certified TestStand Developer says there are at least 6 ways to do anything in TS.)

So, TS is very appropriate for customizing sequence-based activities, and that feature alone is one major reason we use TS at Viewpoint. If we deem it useful for one of our clients or they want to do something non-standard, we will usually find those features in the following list.

• Desired report generation style and content is different than standard TestStand. This desire occurs frequently.
• Database results logged that need to connect to existing databases. This desire occurs frequently.
• Very repetitive use of one or a set of test steps make efficient use of a custom Step Type.
• Special sequence-centric user interface.
• New process model and/or callbacks so changes made once apply across the entire company. This desire occurs frequently with, for example, custom ID scanning for the UUT.
• Special user management interface and login/logout procedures.

So, there are many sequence-centric, even test application specific, features that are addressed with TestStand.

VSI TEC

The VSI TEC was developed to maximize test-code reuse and simplify sequence maintenance for sequences consisting of a linearly executed set of such steps. Consequently, VSI TE is step-centric.

Each step is object-oriented with various methods, such as Edit, Execute, and Render. The creator of a test step must follow a certain API to plug into the TEC environment. The code implemented for those methods has complete flexibility on how the test step behaves.

The VSI TEC test sequencer does not support branching or subsequences, only a list of steps that executes linearly. However, the simple sequence execution is enhanced by other features not presently offered by TestStand. Some of those features are:

• A built-in Hardware Abstraction Layer (HAL) to allow new measurement devices without rewriting the test step code.
• Drag & drop sequence editor with double-click step configuration editing, so one step can be configured to execute in many ways.
• Pre-developed measurement-specific steps for building many standard test sequences, such as measure voltage along a specified switch path.
• Reporting of result data is not tied to any specific data file format.
• Statistical population analysis tied to the sequence ID allowing quick review of all results from a specific sequence.
• Device history for any sequence executions tied to the same UUT ID.
• Restart in the middle of partially completed sequences, in the case that the UUT needs adjustment after failing a step or the test equipment glitches during a week-long test sequence.
• Sequence release handling to incorporate validation procedures.
• File integrity support through MD5 verification, so that, for example, unexpectedly modified steps will prevent execution of the sequence.
• Calibration support to prevent testing past an instrument’s calibration date.

All these features are very important to our clients that want sequence creation to be done by test technicians, test steps development to be done by test engineers, need to track the release and usage of test sequences across multiple test stations and parts results, and have a test application that provides support for test system maintenance and result data handling with out-of-the box capability.

Here’s a screen shot of the user interface to access many of these features, with buttons not appearing unless user access rights allow.

Figure 1 - VSI TEC Utilities

While the VSI TEC is written entirely in LabVIEW, it is supplied as a bundled executable or as licensed source code, because our customers want the option.

Conclusion

NI’s TestStand and Viewpoint’s own VSI TEC have different centricities: one is sequence-centric and the other is step-centric. Both are Test Executives and both have appropriate and potentially non-overlapping use cases. Such variety is available in other TEs, so choose well.