False Positives and Negatives
The equipment measuring a UUT has a known accuracy and precision based on the results of the equipment calibration. Equipment with multiple measurement ranges typically has different accuracy and precision for each range as a percent of range. These variations combine to make any specific measurement uncertain. Look at the situation where the measurement equipment measures a value close to a UUT pass/fail limit. Sometimes, due to variations in the measurement equipment, the UUT performance appears to fall outside the limits, even though the UUT actually performed within the limits. In other words, the UUT will fail the test, even though it should have passed, resulting in a false negative. The opposite can also happen when the UUT is measured as being within the limits even though the actual performance is outside the limits, resulting in a false positive. These incorrect determinations are caused by the equipment measurement variations or inaccuracy and are called Type 1 (false negative) and Type 2 (false positive) errors, respectively, in the statistics world. Figure 1 below shows the issue.
Figure 1. Measurements and part variations
Figure 1 shows the measurement variation at the low and high test limit locations but measurement variation occurs everywhere and becomes especially interesting and important at the edges.
Recall Scenario 1 above, where the high test limit in Figure 1 is 100 ns for a pulse rise time. The equipment, which has an accuracy of ±5 ns, measures 99 ns. Does the test pass or fail? A measurement of 99 ns with an accuracy of ±5 ns means the actual reading is really anywhere between 94 ns and 104 ns. Typically, the test engineer might think that using a bell curve (also known as a normal or Gaussian distribution) is reasonable and that the reading is “probably” 99 ns but, as Figure 1 illustrates, the actual value could be higher or lower.
To make matters more complicated, equipment vendors are not required to provide readings with variations that follow a bell curve. In fact, variations may not be equally distributed. Some equipment may be biased in one direction and other equipment may properly trend to the nominal reading value. If your measurement is close to the test limit and especially if readings are biased low, then it is possible that the equipment reads 99 ns when the unit is actually producing, say, 102 ns.
One method to handle this pass/fail uncertainty is to stay away from these limits by using “guard bands” within the limit range. These guard bands would be proportional in width to deviations in measurements caused by accuracy and precision tolerances.
So, in Scenario 1 above, you should not pass the UUT when the rise time is within 1 ns of the specification limit but the measurement device has a calibrated accuracy of ±5 ns. In this case, a specification limit of 100 ns would need equipment measurements of 95 ns or less for the test to prove conclusively that the unit passed. Note, “conclusively” means 99.7% certainty when the ±5 ns equals ±3 standard deviations (SDs) of the equipment variations. Other SDs will give different certainties.
Figure 2 shows the same product test limits with the measurement accuracy accounted by providing production test limits. The production test limits illustrate that guard bands reduce the measurement error that falls outside the product limits. Note, however, that using guard bands bring up an additional issue because now you are constraining the design to meet a 95 ns limit so that your design can meet your specification. This situation may not be desirable or possible if your unit naturally exhibits behavior close to the test limits.
Figure 2 – Production limits using guard bands to reduce false positives
The popular Six Sigma approach tries to reduce variations in the product so that the part variations become narrower. Thus, without moving the test limits or adding guard limits, there are less (hopefully many less) Type 2 (and Type 1) errors. This concept is illustrated in Figure 3 which shows a narrowed part variation bell curve. We’ve also included an offset (also called bias) shifting the center of the part variations towards the upper limit, as can happen in actual production.
Figure 3 – Six Sigma reduces part variations
In general, you should be considering that there are 4 types of measurement outcomes, as indicated in the 2×2 chart in Figure 4 from Reference 2.
Figure 4 – Various actual pass/fail outcomes and their causes
Generally, only test cases 1 and 2 are considered. Nevertheless, outcomes indicated by the other test cases are possible. We’ve been discussing the Type 2 errors in the upper right of this chart. The goal of any test system is to reduce these Type 2 errors. The test case 3 is also of interest since it causes internal diagnostic and rework when none is needed and the associated costs could be avoided.