Viewpoint News, October 2010

Gauge Repeatability and Reproducibility

by James Campbell - jac@viewpointusa.com

Over the last few months, our accuracy and precision theme has ranged from variations due to measurement instrumentation to variations due to test operators. See the July and September articles for details.

This month will combine all these variations and discuss a method to separate the contributions from the test equipment, the test operators, and the product itself. The method is called Gauge R&R and is a common tool for statistical analysis of the performance of a test system. This method can be fairly easy to use; you don’t have to be a statistician!. Also I discuss how to analyze the data using standard LabVIEW VIs.

Where Do All the Variations Go?

Now that you have the Test System built or refurbished, the measurement equipment is all calibrated and accurate, the Test Operators are trained, and the products are ready to be tested, you need a good way to test the entire assemblage of equipment, operators, and parts to be sure that the results measured on a part are meaningful. In other words, the variability of a part measurement should be mainly due to the part itself and the variability contributed by the Test System and Test Operators should be proportionally small and understood.

The Gauge R&R (GRR) method separates the total system variations, represented as variances (i.e., standard deviation squared, and denoted as s2), into four components as follows.

1. First , the total variance, s2_t, is split into two parts s2_part and s2_meas.
2. Next, the s2_meas is split into s2_repeatability and s2_reproducibility, where:
1. The repeatability component is due to the variability seen by a single Test Operator measuring a single part several times (i.e., the Test System measurement error) and
2. The reproducibility component is due to multiple Test Operators measuring a single part several times.

The s2_meas term is also called s2_gauge and is the variance due to using the gauge, i.e., the Test System. The job of a GRR analysis is to extract the s2_part and s2_meas variances from a set of measurements on multiple parts by multiple operators.

The desired outcome is a small s2_meas variance as compared to the overall s2_t variance. Then, the Test System can be said to be precise enough to handle valid measurements of the products.

Note that the GRR method needs to be applied to each measurement used to determine pass/fail criteria that the Test System performs in the test sequence. Also, a complete GRR needs the specification limits, commonly denoted as LCL and UCL, to give estimates on the probability that the system will produce false positives (bad parts passed) and false negatives (good parts failed).I’m not dealing with the spec limits here, since the understanding of the GRR method basics will make it much simpler to understand the use of the spec limits.

Analysis Methods

The two methods I’ve seen used for GRR are the Average and Range method and the 2-way ANOVA (ANalysis Of Variance).

Average and Range

The Average and Range method computes contributions of the variations using ranges while the ANOVA uses statistical variances. The Average and Range method is (was?) used heavily by the automotive industry (AIAG) and separates the contributions by analyzing the ranges and averages of several operators measuring multiple parts.

There are many Excel worksheets using this method that are available on the Internet. For example, see http://kollmorgen.com/website/common/docs/Gage_R_R.xls, which appears to follow an AIAG standard. Note that I’ve not tried this XLS, so I don’t know if it actually functions correctly.

ANOVA

The other method uses traditional analysis of variance to separate variances into the repeatability and reproducibility contributions. A typical GRR needs a 2-way (or 2-factor) ANOVA. There are many variations on the theme, depending on whether the factors are fixed and not random, but these are subtleties best left to the statisticians. In practice, you usually only need the ANOVA labeled as a random 2-way (or 2 factor) with replicates.

There are many references that discuss ANOVAs, most of which are general purpose unrelated to GRR methods. The general idea of an ANOVA is to separate variations due to factors from any “leftover” or residual variations. In GRR, these “leftover” variations are due to the errors in the Test System measurement equipment, since any other variations are due to the part or operator variations. The factor variations are derived by averaging across all the data associated with each factor value and looking at the variations within each factor value and then between factor values. It’s fairly easy to visualize what is happening when the ANOVA involves only 1 factor. With 2 or more factors, it is are a bit more challenging but follows the same idea.

Note that the ANOVA method is based on the assumption of normally (i.e., Gaussian) distributed values and tests the null hypothesis that the factors have no effect. These assumptions allow calculations of hypothesis rejection probabilities and confidence intervals. (Remember all this statistics stuff?)

LabVIEW has a VI to do this in this 2 factor ANOVA calculation. It is located in ‘…\National Instruments\LabVIEW 2009\vi.lib\Analysis\5stat.llb\2D ANOVA.vi’ and resides in the palette Mathematics>>Probability & Statistics>>ANOVA. I will use it in an example next.

Example

Data Collection and Processing

Let’s do an ANOVA GRR analysis. The following data is from an actual example we recently ran with one of our customers collected from a Test System we helped develop.

The data are shown in the next table.

There are 8 unique parts tested by 2 operators 3 times each (3 replicates). The two factors in this 2-way ANOVA are Part and Operator. As you can see, the factor values are denoted by a label: there is nothing numeric about factors since they are simply categories. Letters are used here for the labels, but they are represented as numbers in LabVIEW to feed the ANOVA VI.

The associated data and VI are located on our website at the following link here.

This VI wants the matrix, as read from a file, to have operators down columns and parts across rows. The first row and column are stripped off after reading the file. You also need to indicate the counts of Parts, Operators, and Replications. Note that in some GRR descriptions, the matrix above is described as a 2 by 8, rather than 6 by 8, by saying that each cell in the matrix has 3 replicates.

The 2D ANOVA VI computes the following values in the returned matrix.

where a => parts as factor 1, b => operators as factor 2, ab => cross factor ab, and e => residual error, and SS = Sum of Squares, DOF = Degrees of Freedom, MS = Mean SS, and F = F Value. Read the LabVIEW help for more detail.

Now What?

As described above, the GRR groups the individual factor variances of s2a, s2b, s2ab, and s2e as

    s2_t = s2_part + s2_reproducibility + s2_repeatability,

where

    s2_part = s2a,

    s2_reproducibility = s2b+s2ab, and

    s2_repeatability =s2e.

The estimates of these variances are found with the MS terms as

    s2_e = MSe,

    s2_ab = (MSab-MSe)/n),

    s2b= ((MSb-MSab)/an), and

    s2a = (MSa-MSab)/bn,

where n is the number of replicates, a = number of parts and b = number of operators. Please see the Montgomery reference for details. Also of interest is

    s2_meas = s2_repeatability + s2_reproducibility.

For this example, the calculations show that the part variance, s2_part, is about 0.121 out of a total variance, s2_t, of 0.125, and s2_meas is about 0.0034. Thus the part variations are dominant in this measurement system and the measurement variations from the Test System contribute about 2.7% of the total variance. If you run the VI on this data set, you will see that the ‘operator+part’ variance is about 1/2 as large as the ‘operator-only’ variance, so, while there is some interaction between parts and operators, it is small.

You can also take the F and associated P significance values from the ANOVA to make predictions about the confidence intervals and probabilities of rejecting the null hypothesis. See the Montgomery reference or nearly any college-level statistics book to learn (or remind) about using these parameters.

Rules of Thumbs

Here are some rules of thumbs that I’ve seen over the years to setting up a GRR experiment and interpreting the results. Remember that many of these are based on the assumption of normal (i.e., Gaussian distributed) data.

1. If you think your data is not normally distributed, see a statistician for help with GRR.
2. For kP=number of parts, kO=number of operators, and kR=number of replicates, have at least kP*kO*kR = 30 measurements.
3. A very good GRR has s2_meas/s2_t less than 1%. Note that the Average and Range method reports this as R&R/TV = 10%, but that is because they use standard deviations which are the square root of variances. Thus 1% = 0.01 is the same as sqrt(0.01) = 0.1 = 10%. Some say that if R&R/TV is more than 30% (or about 9% using the variance ratios), then the Test System needs to be improved to be more precise. Also, some people use a similar ratio called P/T (Performance to Tolerance) as 6*R&R/(UCL-LCL) and want this ratio to be 10% or 30% maximum.
4. Make sure you have parts that range across the entire part tolerance (LCL to UCL) and beyond. As an extreme example, the GRR does not work if all your parts are exactly the same because then the variations are due entirely to the Test System and operators and you can’t make solid conclusions about the precision of the measurement equipment to assess pass/fail criteria.
5. Make sure you have at enough parts. Not only do the part performances need to range across the possible tolerance range, but you need a decent sample set for proper confidence on the ANOVA results. This statement is founded on the fact that the numbers returned by the ANOVA are random (random in implies random out) and the confidence limits on these values are more reliable with more measurements.
6. Make sure to use at least 2 and better 3 operators.
7. The discussion above is good for parts and operators as factors. If you have two Test Systems or one Test System with multiple parts nests, then the ANOVA is now a 3-way or more since more factors are involved
8. The ANOVA only makes precision statements. It assumes your measurement equipment is accurate. If the equipment is not accurate, then erroneous biases are introduced and the results are subject to question. Calibration your equipment before performing a GRR.

References

1. "Design and Analysis of Gauge R&R Studies: Making Decisions Based on ANOVA Method", Afrooz Moatari Kazerouni, World Academy of Science, Engineering and Technology, 52 (2009):http://www.waset.org/journals/waset/v52/v52-5.pdf
2. "How to Screw Up a Variables Gage R&R Study and Cost Your Company Money Before Taking the First Measurement", Mike Hammill. Quality Manager, International Group - Americas Region, Moog Inc, 2009 ASQ Conference Buffalo, NY: http://mysite.verizon.net/vzez3rns/sitebuildercontent/sitebuilderfiles/13.pdf
3. "Introduction to Gage R&R Studies The Key to Understanding Measurement Systems", Hank Scutoski & Chander Sekar, Ph.D., CERPROBE CORPORATION, Gilbert, Arizona, Southwest Test Conference, 1998: http://www.swtest.org/swtw_library/1998proc/PDF/T1_Hank.PDF
4. “Design and Analysis of Experiments”, Douglas C. Montgomery, 5th Edition (2001), ISBN: 0-471-31649-0.
5. “Appraiser Variation in Gage R&R Measurement”, Donald Ermer, Quality Progress, May 2006, Page 7: http://www.stat.purdue.edu/~kuczek/stat513/appraiser-variation-in-gage-rr-measurement.pdf.

Conclusion

The Gauge Repeatability and Reproducibility, or Gauge R&R, method is a wonderful tool to assure that your Test System is making precise measurements. This article discusses what the Gauge R&R is and overviews how to interpret its results. In typical situations, you can perform a GRR analysis with the LabVIEW 2D ANOVA VI.