Viewpoint News, September 2010

Test Variations by Test Operators

by James Campbell - jac@viewpointusa.com

In this article, we discuss the implications of interactions that different Test Operators might have with the unit under test (UUT) and the test fixture itself. We would like to believe that, by automating the testing environment and simplifying operations, every Test Operator would interact with the UUT and test fixture in the same way every time. But the reality is that each operator is unique and, even with extensive training, these differences affect product quality, which, as well known by now, lead to hidden costs to your company. Thus, this newsletter discusses the benefits of experienced Test Operators and ways to develop consistent operators.

Operator Effects of Test Results – 4 Reasons

Reason #1: Operator Training

The level of experience and training for a Test Operator directly affects product test results. The most consistent Test Operators will be the ones who use well documented automated tests, have run the test consistently over a long period of time, or have run through many, possibly hundreds, of units. Ultimately, well-trained operators will be aware if anything on the test setup changes and generally will know if the passing unit is really good or not.

Reason #2: Operator Capability

Understanding the operator’s capability is essential to determining if training can overcome any inexperience which may be resulting in poor quality units being delivered or good quality units being failed. It is possible that the test is too complex for any operator, the UUT has inherent flaws that testing cannot overcome, or that no amount of training or testing experience will ever get the operator to an acceptable level.

Reason #3: Operator Education

The operator’s level of education may not be relevant if the person just has a knack for testing a particular UUT. Certainly, the operator needs a basic understanding of the types of testing and equipment usage. However, consistent operators develop a sense for the performance of a UUT and can utilize the test results in ways that a purely numerical pass/fail decision would miss.

Reason #4: Level of Test Automation

If the test is highly automated, leads the operator through the test, and follows very well-written, step-by-step test procedures, then a less educated operator may be able to perform the test well. On the other hand, if the test is poorly written or leaves critical decisions or measurement reading decisions to the Test Operator, then a more experienced/educated operator may be required. If there is little or no automation and the operator is left to determine or read the results from the instruments on their own, then you may expect more variability in the results and chances are that no amount of experience will ever produce consistent results.

Tips to Promote Consistency

Operator consistency is the goal when testing. The measure of the effectiveness of Test Operators is not that they “pass” more units but that the units that they pass are in-fact “good” units. The measure of the most experienced Test Operators is that they usually get more consistent test results than other operators. Variability in UUT performance should not be overwhelmed by operator variability. Here are some tips to reduce such variability, maybe even to zero.

Tip #1: Documentation and Training

As mentioned several times above, having a clear, well-documented, and understandable test procedure is necessary to assuring consistency. Training the operators based on this document is also required, whether the training is done by other Test Operators or in some formal manner. Assure compliance with this document by measuring performance via a Gauge R&R or UUTs with known characteristics.

Tip #2: Cross Training

By cross-training and regularly moving operators from test station to test station, you gain two benefits. First, you might find testing difficulties, such as in the test setup or test equipment, or confusions in the test procedures, all of which have been masked by an experienced operator. Second, you will be raising the general test experience of all operators, which is useful in higher production demand and operator absences. Finally, as the operators communicate, additional test steps, procedure changes, and efficiencies may be achieved.

Tip #3: Assure Good Designs

A Test Operator who seems to be arbitrarily failing a high percentage of UUTs may not be inexperienced or a poor performer but may have found they are testing a poorly designed unit or are testing the unit using a poorly instrumented test system or the pass/fail criteria may be ill-conceived for the UUT’s performance capabilities. Being too close to the limits of equipment and UUT capabilities lead can lead to large variations in outcomes.

Tip #4: Automation

Test Operators will be involved in UUT testing, even with a highly automated tester. Operator interactions shift from intimate understanding of the product to knowledge of the tester and the product. Nevertheless, the more equipment setup steps and pass/fail decisions made automatically, the less operator variability will be included in the overall system variations.

Conclusion

Variations in product test results arise even with experienced operators. The goal is to reduce these variations. Variations magnify anytime a test procedure is knowledge intensive, leaves judgment to the operator, is not run very often, requires complicated test setups, or relies on test equipment that changes regularly or is not well controlled. Ultimately, you trust the Test Operator to run the required procedure and validate that the UUT is performing as expected. Good equipment, automation, and procedures certainly help, but every effort should be made to have the best Test Operators possible.

Finally, thanks again go out to Ken Samuelson for his contributions and suggestions to this article series.