Are your metrics telling you the REAL story?
What’s worse, a calculation error that would lead to wrong information being provided to the user or an obvious type in a Help file? How about a user menu selection that causes the application to crash vs. a form that requires the user to fill in all of the fields when only one field fails an edit?
Easy, right?
So, ‘defect rates’ should not treat all defects the same. The really significant defects should be weighted to stand out. A serious defect should be weighted 5 to 50 times more than an insignificant defect. For example, in Chapter 3 of his excellent book, A Manifesto for 21st Century Information Technology, Bob Lewis uses the example of a lawnmower factory. Defective mower blades are weighted 1000 times greater than a paint defect.
Mr. Lewis provides an algorithm;
Defect Rate = [∑(Importance Factor*Defect)] / (#Lawnmowers/1,000). He notes, “It looks complicated, but all you’re doing is counting some defects more than others. You assign, for example, an Importance Factor of 1 to bad paint jobs and 1,000 to defective blades. If, in one month, you produce 5,000 lawn mowers with 5 bad paint jobs and 1 defective blade, your defect rate for the month will be (1,000 * 1 + 5 ) / 5 = 201 defects per thousand lawn mowers.”
Make sure you make your metrics measure – and draw attention to – the important things.
Here’s a link to Mr. Lewis’ book.
