|
|
|
Issue:
Volume 8, Number 10
Date:
October 2008
From:
Mark J. Anderson, Stat-Ease, Inc., Statistics Made Easy® Blog
Dear Experimenter,
Here's another set of frequently asked questions (FAQs) about doing design of experiments (DOE), plus alerts to timely information and free software updates. If you missed the previous DOE FAQ Alert, see below.
==> Tip: Get immediate answers to questions about DOE via the Search feature on the main menu of the Stat-Ease® web site. This not only pores over previous alerts, but also the wealth of technical publications posted throughout the site.
Feel free to forward this newsletter to your colleagues. They can subscribe by going to http://www.statease.com/doealertreg.html. If this newsletter prompts you to ask your own questions about DOE, please address them via mail to:[email protected].
For an assortment of appetizers to get this Alert off to a good
start, see these new blogs at http://statsmadeeasy.net* (beginning with the most recent one):
—Round and round on how to round
—ahRrrrggg-Squared — Talk Like a Pirate Day
—Battle with the Black Box
—Fantasy football stats tracked with great interest — $100s of millions worth
* Need a feed from StatsMadeEasy to Microsoft's Outlook? See http://office.microsoft.com/en-us/outlook/HA101595391033.aspx.
1. Forum Alert: Stat-Ease offers new interactive web site — the Support Forum for Experiment Design
2. FAQ: On the Pareto chart of effects, what is the difference between the two levels of significance — Bonferroni vs t-Value?
3. Expert-FAQ: Generating a D-optimal design that includes historical data
4. Expert-FAQ: Choosing a model via backward selection
5. Webinar Alert (2nd): How to Plan and Analyze a Verification DOE
6. Book Giveaway: Several copies of "Design & Analysis of Experiments," 6th Edition by Douglas C. Montgomery
7. Info Alert:Topics ranging from simple factorial on auto part to sophisticated mixture design for semiconductor plasma etch
8. Events Alert: October a peak month for Stat-Ease appearances
9. Workshop Alert: Last chance in 2008 to learn about DOE
P.S. Quote for the month (regarding bias): Bacon observes how people who get after an attractive scent cannot be dissuaded from pursuing that path.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Forum Alert: Stat-Ease offers new interactive web site — the Support Forum for Experiment Design
Stat-Ease now offers a Support Forum for Experiment Design (SFED). This interactive web site at http://forum.statease.com offers an opportunity to post questions on statistical aspects of DOE or
programmatic aspects of Stat-Ease software. Consultant Wayne Adams moderates the statistical side of SFED. Tryg Helseth, who developed the Forum, oversees the questions on Design-Ease® and Design-Expert® program use. Check out the Support Forum for Experiment Design. Please do not be shy: Weigh in with your questions or answers!
I am always learning from other experts about statistical aspects of DOE (no surprise, given my purely engineering background). However, it's not uncommon for users of Stat-Ease software to point out features I'd never thought of (not so good — given that I write the tutorials!). Therefore I am very keen to see SFED flourish. Do not worry, I will remind you about this in future DOE FAQ Alerts. I hope to make our Forum a "go-to" for anyone seeking detail on DOE and, in particular, how its enabled by Stat-Ease software.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2. FAQ: On the Pareto chart of effects, what is the difference between the two levels of significance — Bonferroni vs t-Value?
-----Original Question-----
From: Pharmaceutical researcher
"I designed and executed an experiment using your software. Now I am trying to explain the results. I have some questions on the data analysis, in particular the Pareto Chart. This effect selection tool for two-level factorials provides very helpful thresholds for significance — the Bonferroni (always higher) and t-Value (lower). What is the difference? According to the program's help on the Pareto Chart, both thresholds are based by default on the 5% risk level. If so, why would being above the Bonferroni t be almost certainly significant, whereas exceeding the standard t value be only 'possibly' significant?"
Answer (by Consultants Wayne Adams and Pat Whitcomb):
"The significance associated with a t-test is the risk of falsely rejecting the null hypothesis, in other words, declaring an effect as being significant when it really was caused by chance. When analyzing a two-level factorial many effects are tested. For example, a design on four factors (2^4) generates estimates of 15 effects — four main effects (ME), ten two-factor interactions (2FI), four 3FIs and one 4FI. Repeating a 5% error rate on each of the 15 significance tests causes the overall risk of falsely rejecting the null to be greatly increased — exceeding 50%. One may as well flip a coin!
The Bonferroni correction counteracts this problem of multiple testing on a number (n) of effects. It simply requires that the individual significance level be divided by n. So the Bonferroni significance for the individual test in our 2^4 factorial is 5% divided by 15 effects, or 0.333%. This conservative adjustment causes the critical t value to increase greatly. Thus, effects emerging above the Bonferroni limit are far more likely to be real.
Nevertheless, effects that exceed only the standard t value (calculated without adjusting the risk) may still be chosen if the goal of the experiment is screening and subject matter knowledge provides support for carrying them along to the next phase — a verification study. Effects falling below the t-limit should not be carried any further, at least on the basis of this one experiment."
PS. For a case study with screen shots on using the Pareto to select effects, see "Bonferroni Draws the Line on Over-Selection of Effects" at http://www.statease.com/news/news0706.pdf (June '07 Stat-Teaser). For assessing which factors may truly be impacting your system, I find the Pareto chart to be an extremely useful companion to the half-normal plot of effects for deciding on "how many dogs to let in on the hunt." (Thanks to Rob Reul of Isometric Solutions for this metaphor! Or is this a simile? Or maybe it's an analogy. See http://ask.yahoo.com/20030623.html for guidance.)
Statistical details on how to calculate the t-value and Bonferroni can be found in the Second Edition* of "DOE Simplified: Practical Tools for Effective Experimentation." I posted Chapter 3 at http://www.statease.com/pubs/doesimp2excerpt--chap3.pdf so you can see the appendix showing "How to Make a More Useful Pareto Chart" (page 29).
—Mark
*(Mark J. Anderson & Patrick J. Whitcomb, 2007, Productivity Press, NY, NY. Buy at http://www.statease.com/prodbook.html.)
(Learn more about selecting the vital effects by attending our three-day computer-intensive workshop "Experiment Design Made Easy." See http://www.statease.com/clas_edme.html for a description of this class and then link from this page to the course outline and schedule. Then, if you like, enroll online.)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3. Expert-FAQ: Generating a D-optimal design that includes historical data
-----Original Question-----
From: Operations researcher in the aerospace industry
"When generating a D-optimal design, I need the capability to mandate the inclusion of experiments already accomplished, and then generate the rest of the runs D-optimally. Can Design Expert do that?"
Answer (from Stat-Ease Consultant Wayne Adams):
"Yes. Now for how! From the Design Tools menu choose Augment design. Then choose Augment on the pull-down menu. Choose either factorial D-optimal (for categoric factors) or RSM D-optimal (for numeric factors). Edit the model to design for your expected polynomial. Put the runs into a new block (default) and press OK.
Now for the most important part: How to deal with the runs already collected? That depends on what type of factors are present.
If there are one or more continuous numeric factors you can use the Response Surface Tab, which offers the a Historical design option. Specify the minimum and maximum values for the numeric factors and all the levels for each of the categoric factors. Enter the number of runs already completed. Spelling, punctuation and capitalization all count for categorical factors, so make sure the level labels match exactly with what's shown on the data spread- sheet. Then copy and paste the data into the design layout of Design-Expert.
When dealing with all categoric factors, create a D-optimal design on the Factorial tab with X model points (where X is the number of data points currently collected). Enter 0 for both lack of fit and replicates. Complete the design build and then copy/paste your data into Design-Expert.
That gets you back to where we began — how to augment a design D-optimally.
(Learn more about D-optimal design by attending the three-day computer-intensive workshop "Response Surface Methods for Process Optimization." See http://www.statease.com/clas_rsm.html for a description of this class. Link from this page to the course outline and schedule. Then, if you like, enroll online.)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4. Expert FAQ: Choosing a model via backward selection
-----Original Question-----
From: Industrial statistician at special chemical manufacturer
"Mark, As I've been analyzing mixture data here, I've found that sometimes Design-Expert suggests a linear model, but the quadratic terms provide a p-value of 0.1 or 0.15 — nearly significant. In such cases, sometimes I find that if I choose the quadratic model with backward selection, I get a better fit than with just the linear model. Is this data snooping, or is it a legitimate approach?"
Answer:
Debatable, I think. However, I confess to having tried this approach myself: If higher-order terms can be estimated without any aliasing and they are close to the threshold p value, try the backward regression, which then will likely produce at least one extra term above-and-beyond the suggested model. This may or may not prove useful, that is, very possibly it will be over-fitting the data. Only time can tell via follow-up confirmatory runs.
Stat-Ease Consultant Wayne Adam weighs in with:
"I don't want to step on toes, but as a corollary to your recommendation take a look at the adjusted R-squareds at the bottom of the fit-summary for the various orders. I often use backward reduction from the order with the highest adjusted R-squared. Again this is a rule of thumb rather than a proven statistical technique, but often it gives a better fitting model. 'Assume the model is wrong — confirm that it’s useful': Can I share this quote with Box?"
I replied:
It seems to me that in this case, DX suggests two models and then one ought to go backward from the higher-order one.
Stat-Ease Consultant Shari Kraber says:
"I use this approach any time the p-value of the next higher order is less than 0.2 (assuming no aliasing) and often find another significant term. I didn't think this approach was questionable at all, as long as the p-value of the additional term is strong. I've never thought of it as over-fitting, but instead I think that the group p-value used in the Fit Summary waters down the individual p-values and can easily mask a significant term.
What's Stat-Ease Consultant Pat Whitcomb’s take on this? He says:
"As the models get larger there is more of a chance that there are significant terms hidden in the next higher order model. Remember Pareto; it may be that there are only a few vital terms (and many trivial others). Looking at the whole, the trivial many (insignificant terms) may dilute the vital few (significant terms) and make the group look insignificant. I use the adjusted R-squared as a guide. If the adjusted R-squared increases by a practical amount with the addition of the next group of terms, I add the higher order terms and look at the reduced model, even if additional terms are not significant as a whole."
(Learn more about mixture modeling by attending the three-day computer-intensive workshop "Mixture Design for Optimal Formulations." For a complete description of this class, see http://www.statease.com/clas_mix.html. Link from this page to the course outline and schedule. Then, if you like, enroll online.)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5. Webinar Alert (2nd): How to Plan and Analyze a Verification DOE
You are invited to attend a free web conference by Stat-Ease Consultant Shari Kraber, who will show "How to Plan and Analyze a Verification DOE." This free conference, which Shari will keep at an intermediate level statistically, will be broadcast on Wednesday, October 29 at 12 PM noon USA Central Daylight Time (CDT), which is 17:00 in Coordinated Universal Time (UTC). (We are at UTC -5 under CDT.) She will repeat her webinar at 8 PM that evening (01:00 UTC October 30). The talk will be offered one last time on Thursday, October 30 at 8 AM (13:00 UTC). Stat-Ease webinars vary somewhat in length depending on the presenter and the particular session -- mainly due to breaks for questions: Plan for 45 minutes to 1.5 hours, with 1 hour being the target median.
When developing these one-hour educational sessions, our presenters often draw valuable material from Stat-Ease DOE workshops. Attendance may be limited, so sign up as soon as you see your way clear by contacting our Communications Specialist, Karen, via [email protected]. If you can be accommodated, she will send you the link for the WebConnect and dial-in for ConferenceNow voice via telephone (toll-free access extends worldwide, but not to all countries).
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6. Book Giveaway: Several copies of "Design & Analysis of Experiments," 6th Edition by Douglas C. Montgomery
(Sorry, due to the high cost of shipping, this offer applies only to residents of the United States and Canada.) Simply reply to this e-mail by October 17 if you'd like a free copy of "Design & Analysis of Experiments," 6th Edition by Douglas C. Montgomery. Published in 2004 (John Wiley and Sons, New York), this book remains very useful for learning all the ins and outs of DOE.* Stat-Ease now stocks the 7th edition, thus freeing up 3 copies of the previous one.
I will forward your e-mail entries to my assistant Karen. Do not expect to hear from either of us unless your name is drawn as a winner. However, we do appreciate your participation in these giveaways. Watch for more of these in future DOE FAQ Alerts. Your odds of winning a free book increase by entering each time around!
Reminder: If you reside outside the US or Canada, you are NOT eligible for the drawing because it costs too much to ship the books.
(Montgomery covers DOE completely in his textbook. However, if you seek a hands-on education with a practical focus, attend the three-day computer-intensive workshop "Experiment Design Made Easy." See http://www.statease.com/clas_edme.html for a description of this class,and link from this page to the course outline and schedule. Then, if you like, enroll online.)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7. Info Alert: Topics ranging from simple factorial on auto part to sophisticated mixture design for semiconductor plasma etch
Quality Magazine features a case study showing how DOE eliminated defects at automotive supplier Johnson Controls Inc. The company took advantage of statistical know-how from Middle Tennessee State University (MTSU). See http://preview.tinyurl.com/6ftw48 or click http://www.qualitymag.com/CDA/Articles/Web_Exclusive and look for the link.
Fab Engineering & Operations (FEO) did a great job laying out my article "Mixture DOE for Optimal Plasma Etch" on pages 27-32 of their August 2008 posting at http://www.feomag.com . It's worth pursuing via a free log in. Otherwise, see my manuscript at http://www.statease.com/pubs/mixture_for_optimal_plasma_etch.pdf.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8. Events Alert: October a peak month for Stat-Ease appearances
October is a busy month for Stat-Ease. We started by exhibiting at the Institute of Industrial Engineer's annual Operational Excellence Conference in Minneapolis last week (October 1-2).
This week (October 9) Pat Whitcomb presents "A Factorial Design Planning Process" in Amsterdam to a symposium for life scientists and clinical diagnosticians hosted by Luminex.
Meanwhile, Wayne Adams will head down to Phoenix for the October 9-10 Fall Technical Conference co-sponsored by the American Society for Quality (ASQ) Chemical & Process Industry Division (CP&ID), and the American Statistical Association (ASA) Section for Statistics and Section for Physical and Engineering Sciences (SPES).
Next week (Monday, October 13) I will be at VinylTech 2008 outside Chicago to do a 1/2 Day presentation on "Response Surface Methods (RSM) for Process Optimization and Product Improvement."
Later in the month, Director of Marketing Heidi Hansel Wolfe will represent Stat-Ease at two conferences in Minneapolis
—Medical Device & Manufacturing (MD&M), October 22-23:
see http://www.devicelink.com/expo/minn08/ for details
—Minnesota Quality Conference, October 27-28:
http://www.mnasq.org/newspages/mqcpub.html
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9. Workshop Alert: Last chance in 2008 to learn about DOE
Seats are filling fast for the following DOE classes. If possible, enroll at least 4 weeks prior to the date so your place can be assured. However, do not hesitate to ask whether seats remain on classes that are fast approaching!
—> Experiment Design Made Easy (EDME)
(Detailed at http://www.statease.com/clas_edme.html)
> November 4-6 (Minneapolis)
> December 9-11 (Dallas, TX)
—> Mixture Design for Optimal Formulations (MIX)
(http://www.statease.com/clas_mix.html)
> October 21-23 (Minneapolis)
—> Response Surface Methods for Process Optimization (RSM)
(http://www.statease.com/clas_rsm.html)
> December 9-11 (Minneapolis, MN)
—> DOE for DFSS: Variation by Design (DDFSS)
(http://www.statease.com/clas_ddfss.html)
> November 11-12 (Minneapolis)
—> Designed Experiments for Life Sciences (DELS)
(http://www.statease.com/clas_dels.html)
> November 18-20 (Minneapolis)
See http://www.statease.com/clas_pub.html for complete schedule and site information on all Stat-Ease workshops open to the public. To enroll, click the "register online" link on our web site or call Elicia at 612.746.2038. If spots remain available, bring along several colleagues and take advantage of quantity discounts in tuition. Or, consider bringing in an expert from Stat-Ease to teach a private class at your site.*
*Once you achieve a critical mass of about 6 students, it becomes very economical to sponsor a private workshop, which is most convenient and effective for your staff. For a quote, e-mail [email protected].
PS. We also license workshops to companies with staff that are experts on the application of DOE. Contact Elicia to explore this possibility.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I hope you learned something from this
issue. Address your general questions and comments to me at: [email protected].
PLEASE DO NOT SEND ME REQUESTS TO SUBSCRIBE OR UNSUBSCRIBE — FOLLOW THE INSTRUCTIONS AT THE END OF THIS MESSAGE.
Sincerely,
Mark
Mark J. Anderson, PE, CQE
Principal, Stat-Ease, Inc. (http://www.statease.com)
2021 East Hennepin Avenue, Suite 480
Minneapolis, Minnesota 55413 USA
PS. Quote for the month (regarding bias) — Bacon observes how people who get after an attractive scent cannot be dissuaded from pursuing that path:
"The human understanding, once it has adopted an opinion, collects any instances that confirm it, and though the contrary instances may be more numerous and more weighty, it either does not notice them or else rejects them, in order that this opinion will remain unshaken."
—Francis Bacon, 1620
Trademarks:
Stat-Ease, Design-Ease, Design-Expert
and
Statistics Made Easy are registered trademarks of Stat-Ease, Inc.
Acknowledgements to contributors:
—Students of Stat-Ease training and users of Stat-Ease software
—Stat-Ease consultants Pat Whitcomb, Shari Kraber and Wayne Adams
(see http://www.statease.com/consult.html
for resumes)
—Statistical advisor to Stat-Ease: Dr. Gary Oehlert (http://www.statease.com/garyoehl.html)
—Stat-Ease programmers, especially Tryg Helseth and Neal Vaughn
(http://www.statease.com/pgmstaff.html)
—Heidi Hansel, Stat-Ease sales and marketing director, and all the remaining
staff
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Interested
in previous FAQ DOE Alert e-mail newsletters?
To view a past issue, choose it below.
#1 Mar 01,
#2 Apr 01,
#3 May 01,
#4 Jun 01,
#5 Jul 01
, #6 Aug 01,
#7 Sep 01,
#8 Oct 01,
#9 Nov 01,
#10 Dec 01,
#2-1 Jan 02,
#2-2 Feb 02,
#2-3 Mar 02,
#2-4 Apr 02,
#2-5 May 02,
#2-6 Jun 02,
#2-7 Jul 02,
#2-8 Aug 02,
#2-9 Sep 02,
#2-10 Oct
02, #2-11
Nov 02, #2-12
Dec 02, #3-1
Jan 03, #3-2
Feb 03, #3-3
Mar 03, #3-4
Apr 03, #3-5
May 03, #3-6
Jun 03, #3-7
Jul 03, #3-8
Aug 03, #3-9
Sep 03 #3-10
Oct 03, #3-11
Nov 03, #3-12
Dec 03, #4-1
Jan 04, #4-2
Feb 04, #4-3
Mar 04, #4-4
Apr 04, #4-5
May 04, #4-6
Jun 04, #4-7
Jul 04, #4-8
Aug 04, #4-9
Sep 04, #4-10
Oct 04, #4-11
Nov 04, #4-12
Dec 04, #5-1
Jan 05, #5-2
Feb 05, #5-3
Mar 05, #5-4
Apr 05, #5-5
May 05, #5-6
Jun 05, #5-7
Jul 05, #5-8
Aug 05, #5-9
Sep 05, #5-10
Oct 05, #5-11
Nov 05, #5-12
Dec 05, #6-01
Jan 06, #6-02
Feb 06, #6-03
Mar 06, #6-4
Apr 06, #6-5
May 06, #6-6
Jun 06, #6-7
Jul 06, #6-8 Aug 06, #6-9 Sep 06, #6-10 Oct 06, #6-11 Nov 06, #6-12 Dec 06, #7-1 Jan 07, #7-2 Feb 07, #7-3 Mar 07, #7-4 Apr 07, #7-5 May 07, #7-6 Jun 07, #7-7 Jul 07, #7-8 Aug 07, #7-9 Sep 07, #7-10 Oct 07, #7-11 Nov 07, #7-12 Dec 07, #8-1 Jan 08, #8-2 Feb 08, #8-3 Mar 08, #8-4 Apr 08, #8-5 May 08, #8-6 June 08, #8-7 July 08, #8-8 Aug 08, #8-9 Sep 08, #8-10 Oct 08 (see above)
Click here to add your name to the DOE
FAQ Alert newsletter list server.
Statistics Made Easy®
DOE FAQ Alert ©2008 Stat-Ease, Inc.
All rights reserved.
|
|
|
