Overall Evaluation Criteria (OEC)

The need for an overall evaluation method arises when there are more than one objective that a product or process is expected to satisfy. Situations of this nature are more common in many areas of our lives. Consider the educational system of rating students. Students are evaluated separately in each of the courses they take. But for comparison purposes, it is the Grade Point Average (GPA) which is used most often. Do you ever wonder where the performance numbers like 5.89, 4.92, etc. come from for the Figure Skating competition? Obviously these numbers are averages of all the judges' scores on a scale of 0 to 6. Each of the judges, of course, use the same range of evaluation numbers to evaluate different aspects of the performance like style of skating, how high they jump, how well they land, etc. The scores one judge assigns to a performer come from averaging his/her scores in all separate criteria of performance. Such use of an average or overall criteria of evaluation is quite common in many activities. It is not so common in engineering and science however. (Read an article of OEC, click here)

If an overall index is so common in sporting, educational, and social events, why is it so rare in engineering and scientific studies? It so primarily, because of three difficulties:

Units of Measurements - Unlike GPA or Figure Skating, the criteria of evaluations in engineering and science are generally different (say psi for pressure and inch for length, etc.). When units are different, they cannot be combined easily.
Relative Weighting - In formulating the GPA, all courses for the student are weighted the same. This is generally not the case in scientific studies. Consider, for example the case of baking POUND CAKES. There might be three objectives such as TASTE, MOISTNESS, and SMOOTHNESS that are important. But these criteria of evaluation may not all be of equal importance. How do you decide which one is important? This is generally done subjectively, by project team consensus.
Sense of the Quality Characteristic (QC) - Quality characteristic indicates the direction of desirability of the evaluation numbers. Depending on the criteria and how it is measured, QC can be BIGGER IS BETTER, SMALLER IS BETTER, or NOMINAL IS THE BEST. For example in the game of Golf, a SMALLER(=QC) score is better. In Basketball, on the other hand, a larger score is BETTER(=QC). Again, unlike the sporting and educational events, the difference of QC in scientific studies is quite common. Unless the quality characteristics of different criteria are the same, the evaluation numbers cannot be readily combined.

Example of Criteria with Different QC: Determine the best player

                   Player 1     Player 2      QC
----------------------------------------------------------------------------
Golf (9 holes)     42              52      SMALLER (score)

Basketball         28              18       BIGGER (score)
----------------------------------------------------------------------------
Total Score =    70             70

Are these two players of equal caliber as the total scores suggest? Is the additions of the scores meaningful and logical? Obviously the total of scores have no meaning, as these players to do not perform equally. To make the total score meaningful, one of the numbers must be adjusted such that all QC's are aligned one way (either BIGGER or SMALLER).

A logical and meaningful way to combine the two scores will to first change the QC of the Golf score by subtracting from a fixed number, say 100 (an arbitrary number, expected highest possible for 9 holes of Golf), then add it to the score of the Basketball. The new total score then becomes:

Total score for Player 1 = 30 + (100- 45) = 85

Total score for Player 2 = 20 + (100- 55) = 65

The number 85 and 65 now indicate the relative merit of the players, Player 1 ( 85 ) Player 2 (65).

Multiple objectives are quite frequent in engineering applications. No matter the project, be it a product optimization, process study, or problem solving, the desire to satisfy more than just one objective is quite common. Because the criteria involved are different, the experimental results (be it DOE/Taguchi or otherwise) are generally analyzed one criteria at a time. This approach, of coarse, does not guarantee that the best design obtained for one criterion, will also be desirable for the other criterion. What is needed is a properly formulated Overall Evaluation Criteria (OEC) number representing the performance of the test sample. Thus, when there are multiple criteria of evaluations, lack of such formulation poses a major hurdle for analysis of DOE results.

OEC Formulation
When the product or process under study is to satisfy more than one objective, performances of samples tested for each trial condition are evaluated by multiple criteria of evaluation. Such evaluations can be combined into a single quantity, called Overall Evaluation Criteria (OEC), which is considered as the result for the sample. Each individual criteria may have different units of measurements, quality characteristic, and relative weight. In order to combine these different criteria, they must first be normalized and weighted accordingly as shown below.

The method of OEC formulation and computation can be studied by considering the cake baking.

                         EVALUATION CRITERIA DESCRIPTION
Criteria            Worst Value (w)     Best Value (b)          QC            Weighting (Rw)
1.Taste                       0                         12                               Bigger             55%(Rw1)
2. Moistness               25 gm                   40 gm                         Nominal           20% (Rw2)
3. Consistency             8                          2                               Smaller            25% (Rw3)

The evaluation criterion were defined such that TASTE was measured on a scale of 0 to 12 (bigger is better), MOISTNESS was indicated by weight with 40 gm (target value) considered the best value (nominal is the best) and CONSISTENCY was measure in terms of the number of voids seen (smaller is better).

Assume that the cake sample for trial#1, the readings are (T, M, C):

Taste T = 9, Moistness M = 34.9, and Consistency C = 5

Then OEC for the cake sample is can be expressed as:

OEC = [ |(9-0)|/(12-0) ] x Rw1 + [ 1 - |(34.9 - 40)|/|(40-25) ] x Rw2 + [1 - |(5-2)|/(8-2)] x Rw3

Explanation of data reduction:

Numerator (9 - 0 ) represents (reading - worst value) in case of BIGGER QC

Numerator (40 - 34.9 ) represents ( reading - target value) in case of NOMINAL QC

Numerator (5 - 2 ) represents (reading - best value) in case of SMALLER QC

Denominators (12 - 0, 40 - 25, and 8 - 2) represent differences between the best and the worst values for all QC. The worst value in case of NOMINAL is the worst deviant of the data extremes from the target.

Note: Before all criteria of evaluations can be combined, their QC's must all be the same. The second expression is modified to change the NOMINAL QC, first SMALLER by finding the deviation from the nominal, then to BIGGER. The third expression is modified to change the SMALLER QC to BIGGER. The numerator in each term is calculated by subtracting the smaller magnitude (or target value in case of NOMINAL) from the reading, then taking the absolute (|x|) value. The denominator is always the range of data spread, which is positive difference between the best and the worst reading for the criteria.

OEC     = (9/12) x 55 + [1 - (5.81/15)] x 20 +    [1 - (3/6)] x 25

             = 41.25 + 12.25 + 12.50 = 66.00

Since an L-8 orthogonal array described the baking experiment, there are eight cakes baked with one sample per trial condition.. The OEC calculated above (OEC = 66) represents the result for trial#1 . There will be seven other results like this. The eight values (OEC's) will then form the result column in the orthogonal array. The process will have to be repeated if there were more repetitions in each trial condition.

Quality Characteristic for OEC - Depending on the QC of the constituting evaluation criteria, the QC of the OEC can be either BIGGER or SMALLER. I the above case, all criteria evaluations were aligned to be of bigger QC as the first criterion with 55% relative weighting has Bigger QC. All versions of Qualitek-4 prior to version 6.5 allowed QC of OEC to be either Bigger or Smaller as the relative weight of the criteria dictated (automatically set). With the release of Version 6.5 of Qualitek-4 software will always assign Bigger is better QC to all OEC values regardless of the QC of the criteria it is made of. [Download and try new OEC capabilities of Qualitek-4 software from www.rkroy.com/wp-q4w.html ]

Observe that the evaluation (T, M and C) in each case is first modified to show the positive difference between the reading and the smaller magnitude of the best/worst values (or target for NOMINAL), then divided by the allowable spread of the data. This is done to get rid of the associated units (Normalization). Subtraction of the fractional reading from 1, as done for the second and the third criteria, is to change the quality characteristics to line up with the first criteria, that is, BIGGER IS BETTER. Each criteria's fractional value (y/y-max) is also multiplied by the corresponding weighting and added together to produce a net result in numerical terms. The manipulation above normalizes OEC formulation to produce numbers between 0 - 100.

Author of OEC Concept: Ranjit K. Roy, Ph.D.,P.E.

REFERENCES:
1. Design of Experiments Using the Taguchi Approach : 16 Steps to Product and Process Improvement by Ranjit K. Roy. Hardcover (January 2001, John Wiley & Sons)

2. A PRIMER ON THE TAGUCHI METHOD by R. Roy.

3. Report# 1. Multiple Criteria of Evaluations for Designed Experiments