Forming Experiment Planning Team for Brainstorming

• Appoint Team Leader
• Form Team
• Appoint (or self-appoint) Facilitator
• Schedule Planning Meeting

Your project team (3 - 12 people) should consist of project stakeholders and people with first hand knowledge about the project. You should attempt to form a team and proceed with experiment planning following the guidelines provided below.

Carrying out an experiment is same as completing a project. Like a project, your experimental effort will have a team leader and project team. For coordinating discussions and information gathering, your experimental project will also benefit from an independent facilitator. Of course, for small projects, you or project responsible individual can play all three roles (leader, team, and facilitator). Regardless of the project size and whether different individuals perform these roles or not, it would be helpful to carefully examine the tasks involved in these roles.

Identify and fill these three positions based on the roles and tasks they need to accomplish:

• TEAM LEADER – It not you, it is some body who is responsible to carry out the experimental project. The team leader’s main interest is to get people with first-hand knowledge together and collect information about running the experiments as quickly and as objectively as possible.
• TEAM MEMBERS – Those who have the first-hand information make better team member. The ideal team size is between 3 -12. Team leader should form the team and schedule a day (4 -8 hours) for experiment planning session. Team members need not have any special preparation or DOE background to be part of the team.
• FACILITATOR – A a facilitator needs to be an experienced DOE practitioner from among the team members or someone uninvolved in the project. He/she must have background in DOE, for without it, he/she will not be able to guide the team toward an effective application. This individual must be FAIR and OBJECTIVE. To be fair is to avoid any appearance of preferential treatment of any individual participants or an idea proposed. All team members need to be treated equal. When soliciting opinion, each individual gats the same weight. To be objective is to avoid opinion as much as possible and demand data in support of decision making process.
• STAKEHOLDERS – These are individuals who care most about the outcome of the project. Your management and all members of the team are stakeholders. By stakeholders, we generally refer to the management personnel with responsibilities for funding and project timing decisions.

Your experiment planning (or project) team should be consulted on selecting a day and location for the planning meeting. You should secure commitment from all team members for the entire day (8 AM – 5 PM).

For efficient use of time, you may breakdown your project team in two different groups.

Project team – Group 1: Team leader and few key people in the project with responsibilities to make decision about the project goals, objectives and implementation. It should include management personnel (stakeholders) with direct responsibilities about the product/process under study.

Project team – Group 2: Team leader and members with direct knowledge about the project who will have clear understanding about how to set up the tests and limitations of the variables to be included in the study.

On planning day, if you are the leader as well as the facilitator, you must be extra caring and conduct the meeting in most professional manner (fair and objective). Seek out information of the following listed items by involving everyone and by consensus when appropriate.

Selecting Project Title

• xx
• Agree on a Title for the experimental study

Your project title is an important identifier of the activities you are undertaking. It needs to be something that relate to the product or process under study. It should include works that indicate the purpose of the study. Thus if your application involves studying a process that makes a plastic bracket to determine better process parameter, your title may be “Plastic Molding Process Optimization Study”. Since the purpose often is to solve problem or optimize designs, your title should include words like STUDY, INVESTIGATION, EXAMINATION, etc.

To help you and your team think through the process under study, here are some questions for which you are seeking answers.

1. Where in the overall business scheme do you work?

(Research, Design, Validation/Test, Manufacturing, Fulfillment/Deliver, Product Support, Warranty/Loss Prevention)

2. What type of application of DOE are you going to have?
   1. Our DOE application project will focus on:

                                                               i.      process optimization

                                                             ii.      product design optimization

                                                            iii.      problem investigation and solving

                                                           iv.      Validation test layout.)

2.  (if process optimization) What does your process do? 
3. (if product or design optimization) What is your product? What does your product do? 
4. (if problem investigation and solving) What is the nature of problem you are trying to investigate and/or solve? 

                                                               i.      What is it that is considered to be creating problem?

                                                             ii.      What will be absent when you solve this problem?

5. (if validation test layout) What is the product or process you are laying out the validation test for? 
2. What would be the proper title for this project?

Describing Objectives of Experiment

• Describe the purpose of launching the study in 2 – 5 connected sentences.

• Identify & List all objectives (show them separately when there are multiple objectives)

Your reason for the experimental study may be to solve a problem, optimize designs, layout validation tests, or increase response from advertisement, etc. No matter the real reason, you may view the activity as trying to solve a problem that is to fill-in a void or absence of something. In other words, when you are finished with the experiment, you will obtain something that you do not have now (problem). While stating the problem, indicate the benefit you will derive as a justification for doing the project. Here is how the project description can be composed for the PLASTIC MOLDING project indicated earlier.

Example DESCRIPTION:

“We have been experiencing high rejects and warranty from our plastic molding process. This study is undertaken to determine process parameters that will reduce our scrap rate. The improved process design is expected to keep our customers satisfied and affect our bottom line.”

Here are a few questions that will help you collect your thoughts and determine suitable answers.

- Your reasons for performing this project are?

- What is it that you want to accomplish with this project?

- List your objectives/goals for this project 

In case your study involves BAKING POUND CAKES, you may consider the objectives to be to: (a) Improve TASTE, (b) Increase MOISTNESS, (c) Prolong SHELF-LIFE.

Measuring Performance or Results

• Review & refine all objectives

• Define EVALUATION CRITERION for each objective.

In your study, when you run a set of experiments laid out by DOE, you would want to see how each test sample performs in terms of satisfying the objectives. But, how would you know how well a particular objective is satisfied?

Each objective can be evaluated by one or more characteristics called the EVALUATION CRITERION which measures how well an objective is satisfied. For example, an evaluation criterion like “a subjective scale in the range of 0 – 10” can be used to evaluate TASTE of cake. Where as, the criterion “weight in grams” could be used to evaluate MOISTNESS of cake. Consider the case of a study involving heat treatment of a steel part in which one of the objectives was to improve strength. The criterion of evaluation for the strength, in this case, was “tensile stress”. (Just identify the name. The units of measure and range for these criteria of evaluations will come later). Repeat this process for all objectives.

Helpful task and question:

• List the performance objectives

• For each objective ask, “How will the performance of this objective be measured?”

Identifying Units of Measurements

• Indicate the units of measurement for each criterion and identify their range of measurements.
• List all evaluation criteria and their units of measure in a table.

The criteria of evaluations you use to measure how well each objective is satisfied may be one that is a standard practice in the industry, or defined uniquely for your results. They may also be subjective or objective. If any standard unit of measure for a criterion is absent, you will need to establish the units of measure (lbs, inches, grams, numeric, etc) and its expected range (example: 0 – 8 for numeric, 200 – 300 psi for pressure, etc.). Consider each criterion and determine their units and range. Repeat this process for all criteria. Remember, all criteria of evaluations must be in quantitative terms.

Helpful tasks and questions:

• List the criteria of evaluations
• For each criterion, ask “What are the units of measure?”

1. Is it Numeric/non-numeric or Subjective

                                                               i.      If Numeric = “What is the unit of measurement? [examples: temperature, %F or %C; concentration, %; quantity (any number)]

1.       Establish range (low & high values) for each of these measures that you expect the experimental results to fall within.

                                                             ii.      For analysis of DOE results, you must have a numerical value for evaluations. For subjective evaluations, select a value scale that could be used for measurement?” Options given: 0-to-5, 0-to-10, 0-to-100, etc. Select higher ranges (0 – 20, 0 – 100, etc) only when you are able to discriminate between two successive numerical evaluations (say between 19 & 20)

2. If you were to run several experiments on this project, what would you expect to be the worst and the best results? Your answer must fall within range identified earlier.
3. Within this range specified, is there a target value you want to achieve? If yes, what is that value? In case you have such a target value, it will be considered the best value in this evaluation. For example, in case of evaluation of MOISTNESS in cake baking experiment, the weight of a fixed size of cake may be measured as an indication of moistness. The weights 25 grams – 70 grams may be the range, but 40 grams may be most desirable (from past experience). In this case 40 grams become the best result.
4.  How do you plan on measuring your second objective? Repeat (a) through (c) above.

Determining Evaluation Criteria and Creating OEC Table

• Complete the evaluation criteria table

• Formulate OEC equation, if applicable and desired

• Note QC for OEC (Always bigger is better in Qualitek-4).

Having identified all objectives and their respective criterion of evaluation, you are in a position to determine their individual quality characteristic (QC) and compile the information in the following table. 

After all criteria are listed and their QC identified, you will proceed to prioritize their importance in terms of relative weight. You may easily do so by allowing each member of your project team (also true when you are alone in the team) to distribute 100 pennies ($1) among all the criteria. In doing so, one must distribute all (100 pennies) to the criteria as per the importance in his/her own mind. After all team members are done assigning pennies to the criteria, add all pennies assigned to a criterion and divide it by the number of participants. The number you obtain, represents the group consensus on relative importance of the criterion. (Make sure the sum of percentage weights of all criteria always add up to 100) 

Example: Cake Baking Process Study 

1. Taste                                  0                        10           B               55    

2. Moistness                          25 gm                   50gm     N                 30

3. Smoothness (#voids)           6                          0            S               15  

When you have multiple objectives, a common practice will be to analyze results under each criterion separately. Such separate analyses are then expected then to produce different factor influences and optimum conditions. It would be a matter of extreme coincidence if all optimum conditions turn out to be the same.

As an improvement over this practice, you may wish to combine results under all evaluation criteria into a single index, and then analyze using this index. The method for combining such multiple objective results in to an overall evaluation criterion (OEC) is described the texts by R. Roy (http://nutek-us.com/wp-txt.html ). For quick refresher on the theory and background, you may visit: http://nutek-us.com/wp-oec.html . Should you want to see how Qualitek-4 handles such results of multiple criteria, select example experiment POUND.Q4W and use OEC Results from EDIT menu and review the method of combining different evaluations into OEC for each test sample.

Brainstorming & Qualifying Factors for Study

• Solicit ideas and prepare a Long List of potential factors.

• Scrutinize all ideas and prepare a Qualified List of factors.

• Paretoize the list of factors (From most important to the least).

 (This discussion & brainstorming is done with entire project team—or team members with first-hand knowledge/responsibility about the process under study.)

7a. LONG LIST

The purpose of this meeting or discussion will be to gather ideas & suggestions about how to make improvement (results). Realize that, by now all involve already know about what you are after and what are the objectives. The goal here is to capture a quantity of ideas and list them. All ideas gathered do not necessarily make valid factors. However, all suggestions & ideas solicited must be collected without concern for validity. The time for scrutiny and consideration for study will come later.

Below are sample questionnaires that may initiate thoughts about factors:

·         What are some of the actions you can take to improve and satisfy performance objectives?

·         What are variables (materials/environmental factors/constituents/settings/parameters/etc.) that may influence the outcome of your project?

·         If you have done some process studies and have prepared cause-and-effect diagrams (fishbone or Ishikawa diagram), what are some of the factors you identified?

If you have a number of people in your project team, this is a good time to ask ideas from each and everyone. You do not want to leave any “stone” unturned.

If you are alone in the project, or working with a few members in your team, it is a good idea to pause and attempt to collect as many ideas as possible. For the preliminary list of ideas, the longer the list the better chance you have to capture all possible influencing factors. 

Be sure to utilize previous research or captured thoughts of team members that utilized FISHBONE/ISHIKAWA DIAGRAM for generating ideas.:

   Manufacturing (6 M's) - Man, MAchine, Method, Materials, Measurements and Mother Nature

  Service (4 S's) - Skill, System, Supplies and Surroundings.

Establishing Levels of Study Factors

• Establish levels of all factors included in the study

• Modify scopes of design (Array size) after levels of all factors are determined.

 The list of factors (A, B, C, etc.) in the Study List now needs to be carefully examined and their levels determined. The first issue in determining the level is to decide how many levels this factor should have. Generally, all factors should have two levels, but may have three or four levels. If a factor is a discrete/fixed factor (like tools, machine, shifts of day, male/female operator, etc.) it may have more than two levels. Also, if a factor is KNOWN to have nonlinear behavior, it may be necessary to study it in three or four levels. Otherwise, you should study all factors at two levels when possible. 

Approach you should follow is to consider each factor separately and determine:

• Number of levels it needs to have (2, 3, or 4—

• Value or description of the levels

Use these guidelines:

• Number of levels should be 2 unless required because the factor is discrete or known to be highly non-linear.

• The values of factor should be as far away from either side of the current working condition as possible. The levels should be such that it provides results that are measurably different, yet it should be economical, practical, and can easily be released if identified as the optimum.

This way, complete determining the levels of all factors in the study list. Should you have a factor that needs to be at more than 2 levels (3 or 4), you will then need to drop factors to make room for this factor (level upgrade) or go for a larger array. For instance, if you happen to have one of the 7 factors to have 4 levels, you will need to drop 2 of the seven factors to make room for a 4-level factor. You will then modify the L-8 array to accommodate this 4-level factor and four remaining 2-level factors. However, before you can complete the design process, you will need to consider INTERACTION and NOISE factors that might be part of your study and may indeed reshape your experiment.

9

Selecting Interactions and Noise Factors for Study

• Select any interactions that you wish to study

• Decide if you wish to formally include noise factors in your design and may be able to go for outer array design

• Re-evaluate scope of experiment by determining the experiment design (Inner and Outer array sizes)

Now that you have selected the factors for the study and determined their levels, you need to consider possible interactions that you may want to include in the study. For interactions, consider only the interactions between two 2-level factors (like AxB, BxC, etc.). Understand that, if you have 7 2-level factors, there are possible 7 x (7-1)  = 21 interactions. You are now faced with two questions: how many interactions to study, and which ones among all possible ones to study. Generally, you do not have any knowledge to answer these questions. But, if you happen to have the knowledge and/or conviction to decide upon some interactions to study, you will have to revise your experiment design. Suppose that you have two interactions that you must study. Since your limit on the size of the experiment is 7 columns (in an L-8), you could do so  by discarding two factors to make room for he two interactions that have a common factor. If on the other hand, you want to study all 21 interactions and 7 factors, you will require an array that has 28 or more columns. To do so, you will need to increase the size of the experiment and go for an L-32 array. 

A general recommendation is that, you select the biggest array possible and accommodate all factors first. Then if you have spare columns, reserve them to study interactions. 

The last item to consider before finalizing experiment design is to consider the possibilities of formally incorporating the effects of noise factors in your experiments. The most desirable way to include uncontrollable/noise factors in your design is to go for an outer array design where an orthogonal array is used to formally combine the noise factors to create some critical conditions. To select robust design conditions, the tests under different recipes of the control factors are tested by exposing them to the influence of the noise condition so created. The noise factors, of course, are uncontrollable in real life, but are assumed to be controllable while conducting the tests under laboratory environment.

If there are 3 noise factors in your experiment, you would use an L-4 array as an outer array. This will require that you run each trial condition (of the control factor) 4 times by exposing them to 4 separate noise conditions. Such formal treatment of the noise factors, require more samples and time in carrying out the experiments, but is likely to produce more useful information about the system under study. 

A general guideline to follow is to go for robust design approach using outer array, if not possible, carry out multiple sample tests in each trail conditions under random noise condition.

10

Indicating Sample Size for the Trial Conditions

• Determine number of samples you wish to test in each trial condition

• Identify, samples in each cell in case you have an outer array.

Before you complete experiment planning discussions, you should know and be able to tell your team about the scopes of the experiments in terms of how many test samples would be involved and how the results should be collected. You could accomplish this by simply creating a table of data collection sheet that shows how many tests (trial conditions) would be done and what results would be collected under how many samples and criteria of evaluations.

11

Preparing Instructions for Conducting Tests

• Prepare instruction for running experiments if any

• Establish responsibilities, location, and completion times  

• Prepare and share experiment planning summary with all team members                                                                                                                                       

Before you adjourn your planning meeting, you need to discuss and share plans for acquiring test samples, test facilities, and data collection procedures with all in the team. If possible, you should also form consensus on the length of time and schedule of completing the study.

Prepare a summary of the information gathered from the planning session. This could be a quick review with the group before you adjourn meeting with the team, or prepare it after the meeting and share it with the team members. You will need this summary page when you start using Qualitek-4 to design the experiment. Your planning summary should contain the following information. 

Project Title ________________________   Location _____________________

Participants:      1.__________________  2.____________________________

                        3.__________________  4.____________________________

Criteria Description           Worst Value     Best Value      QC          Rel. Weighting

1.

2.

3.

etc._____________________________________________________________

Your OEC Equation (if planned)

OEC =   (         ) x     +   (               ) x     +   (                  ) x     +   (                ) x

Example:

FACTORS                                 Level 1              Level 2              Level 3          Level 4

1.

2.

3.

etc._________________________________________________________

List Interactions and Noise factors you wish to include in your study.

Proposed Experiment Design: Indicate the inner and outer arrays used for the experiment design and how the control factors and noise factors will be assigned to the columns of the arrays. Based on the proposed design, indicate the test sample size requirements.

12

Project Report with Experiment Planning, Design, and Analysis

After experiment planning discussion, you will design your experiment (Qualitek-4), carry out all prescribed tests and collect results. You will then perform analysis of results (Qualitek-4) and may wish to compile a report with the findings and recommendations. Your report should contain detailed descriptions in the following areas. 

Report Content (A comprehensive WORD document)  

1.       Project Title

- include names of team members (participants in the study)

2.       Brief Description of the project function and the purpose of the study

- indicate reasons for the study and the benefits derived

3.       Evaluation Criteria 

- discuss how different objectives were measured

4.       Factors and Levels

- explain how factors were selected from a long list

- discuss the rationale for number of levels and their values

5.       Noise Factors and Interactions, if any

- discuss how interaction (if included) was selected

- justify use of noise factors in the design (if included)

6.       Orthogonal Array and the design

7.       Main Effects – indicate trend of influence of factors and interactions

8.       ANOVA – list factors with higher relative influence to the variation

9.       Optimum Condition and Performance (convert if in S/N)

- indicate any factor level adjusted for interaction

10.   Confidence Interval (C.I.)

11.   Expected Savings from the new design (Capture & paste Qualitek-4 screens for items 6 – 11)

12.   Conclusions and Recommendations

- based on the results, what do you propose for further study

- explain when you would know that the experiment is satisfactory