Viewpoint News, February 2011

Part 2: When Is a Test Executive Cost Effective?

by James Campbell - jac@viewpointusa.com

You’ve reviewed the outline last month of pros and cons of using a Test Executive, and you decided that the reuse and leverage benefits apply. But, you wonder how you can justify the extra development costs. How many test systems do you need to develop and install before you start to see the cost benefit ROI of the leveraging?

Cost Analysis

What is the cost difference between developing test systems with and without a Test Executive? Let’s run through a hypothetical cost analysis to illustrate the important factors. You can also look at another calculator provided by NI at: http://sine.ni.com/apps/utf8/nicalc.ni?calcid=1001&area=main&func=display.

Custom Test Applications

Here’s a high-level list of costs for developing custom test applications.

Assumptions:

• A competent Test Engineer will do the development. This person is paid a salary of 80 k$ per year (which costs the company about 100 k$ after including benefits, overhead, taxes, and so on). My observation is usually that a less expensive (i.e., less experienced) Test Engineer will take longer to develop and debug an application than an experienced Test Engineer, so the final overall cost will be at least equivalent and likely more in both cases. This 100 k$/year works out to about 50 $/hour. (Believe it or not, the actual cost to the company is about 80 to 90 $/hour for a fully loaded when management overhead, training, vacation, sick time, floor space, computer, desk, and other operational costs are included. But the higher wage just makes the case for a Test Executive even stronger, since fixed costs are proportionally smaller.)
• I want a no-frills application. I’ve not allowed any time for building tools that would be reused in future applications. Why? If I do, then I might be starting to write my own TE, and that’s not allowed in this exercise by definition.
• There are 4 measurement devices, which need drivers written and tested. Let’s allot 6 days (1.5 days per device).
• LabVIEW is the development tool and is well known by the developer.
• 5 Test Stations, each one with different product tests, so there is little commonality in test steps.
• Documentation for the application consists of a simple User Manual and a simple Theory of Operation document. Just enough to hand hold the Test Technicians using the application and remind the developer what he did a year from now when changes are needed. Say 5 days.
• The steps are not very complex.
• The product has 10 test steps and each one takes 3 days to understand the specs, write the step test code, test the step execution, and integrate into the app.
• The features of the test app itself take 4 weeks (20 days) to develop from scratch which covers designing the app UI, deciding on configuration and data file content, and other application specifications, writing the app, testing the app, integrating all the test steps, and performing final debug and testing. Finally, there is an additional 5 days to work out all the kinks when used on the real Test Station, because the hardware never works exactly like you expect. This effort sums to 25 days of work.
• Maintenance time per Test Station to address (and fix?) application issues take 10 days per year. This effort does not include feature upgrades, which would be significantly more costly.
• The lifetime of the Test Stations is 10 years.
• It’s OK to use rough numbers in the cost calculations.

The total time to develop this test application is then 6 (drivers) + 5 (docs) + 55 (app and steps) = 66 days. There are surely details amounting to 10% that I’ve not included or the developer did not consider. So, multiply all the estimates by 1.1, so the total is about 73 days. Using the 50 $/hour number and rounding to the nearest 1 k$ gives the following table.

Summary:

So, the total costs would be:

• 145 k$ for the 5 stations
• 5 k$ for one-time cost of Development Tools.
• 20 k$ for the 5 stations for annual maintenance cost, which does not include new features and troubleshooting of the Test Station itself.

So, we estimate, for all 5 stations, 150 k$ for development time and tools and 20 k$ for maintaining all per year.

Developing with a TE

Here’s a high-level list of costs for developing the same test applications using a TE.

Assumptions:

• Same assumptions as above, except:
• User Manual is a one-time charge, since all Test Systems have the same User Interface. The Theory of Operation is also a one-time charge for the same reason.
• No need for the 20 days to write the application, since we will just use the Test Executive UI. So, the total test step development is the original 30 days for the 10 steps plus an additional 1/2 day per step for connecting to the TE (5 days total). Finally, keep the same 5 days for debug. The total is 40 days.
• Half the maintenance time per Test Station since you are only dealing with test steps and not the application as well, giving 5 days per year.
• TestStand and LabVIEW are the tools. Other tools cost similarly whereas some can be 5-10 times more because they target enterprise installations.
• Learning TestStand takes a 1-week training class (4 k$ class + 2 k$ time) plus 5 days post-training (2 k$ time) to become familiar.
• Same 10% underestimation of time as above.

Summary:

So, the total costs would be:

• 105 k$ for the 5 stations
• 10 k$ for one-time cost of Development Tools.
• 8 k$ for one-time cost of learning TestStand.
• 2 k$ for one-time cost of Documentation.
• 10 k$ for the 5 stations for annual maintenance cost, which does not include new features and troubleshooting of the Test Station itself.

This estimate sums, for all 5 stations, to 125 k$ for development time and tools with 10 k$ for maintaining all per year.

Conclusion

I analyzed the cost of developing test applications for a simple test system. I split the costs into annual maintenance and upfront on-time costs for development efforts and tools. Let’s look at the upfront costs first.

For all 5 Test Stations, the upfront costs for the Test Executive (TE) approach costs about 125/150 of the custom approach, or about 17% less. Over the 10 year life time of the Test Stations, including the maintenance, the cost ratio is 225/350, or about 36% less than the custom approach. The ratio improves since the annual costs for the TE approach are 50% of the custom approach.

For just 1 Test Station, the TE approach costs about 41 k$ (21 k$ dev, 10 k$ for tools, 8 k$ for learning, 2 k$ for docs). The custom approach is 34 k$ (29 k$ for dev, 5 k$ for tools). So, with only 1 station, it’s cheaper to do the custom route. Clearly, a TE requires multiple uses to justify the initial higher expense, but not by much!

Next month, I’ll supply an Excel worksheet you can use to estimate your own situation.