A NEW PERSPECTIVE ON THE DUNNETT PROCEDURE:FILLING THE GAP BETWEEN NOEC/LOEC AND ECx CONCEPTS
MARIE-LAURE DELIGNETTE-MULLER,*y CAROLE FORFAIT,y ELISE BILLOIR,yz and SANDRINE CHARLESyyUniversite de Lyon, Laboratoire de Biometrie et Biologie Evolutive, Villeurbanne, France
zPole de Recherche ROVALTAIN en Toxicologie Environnementale et Ecotoxicologie, Valence, FranceUniversite de Lyon, Lyon, and VetAgro Sup, Marcy lEtoile, France
(Submitted 21 June 2011; Returned for Revision 17 August 2011; Accepted 8 September 2011)
AbstractThe no-observed-effect concentration (NOEC) is known to be based on a wrong usage of hypothesis tests, and the use ofcondence intervals is preferred. The purpose of the present study is to provide an easy and proper way to interpret ecotoxicological testsbased on simultaneous condence intervals associated with the commonly used Dunnett procedure, and to show how these intervals mayallow one to infer ECx values (effective concentrations). Environ. Toxicol. Chem. 2011;30:28882891. # 2011 SETAC
KeywordsEcotoxicological bioassays Simultaneous condence intervals Toxicity endpoints Multiple comparisons
In ecotoxicology, the effects of contaminants on livingorganisms are usually measured at the individual level, inthe laboratory, and according to standards. This ensures thereproducibility of bioassays and the control of environmentalfactors, often confusing in an in situ approach. These bioas-says,which test acute or chronic toxicity, generally apply toreproduction, growth, or survival of organisms. The mostwidely used analysis of standardized bioassays leads to thedetermination of the no-observed-effect concentration (NOEC)or the lowest-observed-effect concentration (LOEC) on thebasis of null-hypothesis signicance tests as recommended inthe Organisation for Economic Co-operation and Developmentguideline . However, the relevance of these toxicity end-points has often been questioned . From a statistical pointof view, a strong argument against the NOEC is that it relies onan improper use of p values, which are not intended to establishbiological thresholds. Indeed, the NOEC is dened as the higherexperimentally tested concentration for which the difference tothe control is not statistically signicant, but it is widely used inrisk assessment as a zero-effect concentration, which is anincorrect interpretation . In fact, this misinterpretation of pvalues is very common and comes from the wrong assumptionthat absence of evidence is evidence of absence . Manyscientists reported this type of error in various research areasand advocated the use of condence intervals instead of null-hypothesis signicance tests [4,7,8]. Nevertheless, to ourknowledge, no method based on condence intervals has beenproposed in ecotoxicology to replace the NOEC determinationby using a Dunnett procedure, and a current belief seems to bethat no condence interval can be given on the NOEC.
To show that a tested concentration of a contaminant has noor at most a negligible effect on the response to an ecotoxico-logical bioassay means to show that the difference between thistested concentration and the control is null or negligible. This
task is closely linked to that of bioequivalence studies com-monly performed in drug trials, for which a statistical procedureis now well established . To ascertain bioequivalencebetween two drugs for a pharmacokinetic measure, the 90%condence interval for the ratio of the two means should fallwithin a bioequivalence limit corresponding to an acceptablerelative difference of 20%. In fact, one may not state equiv-alence from condence intervals without dening a level ofacceptable difference. In the same way, to show that a testedconcentration of a contaminant has no or a negligible effect onthe response of ecotoxicological bioassays, one needs to pre-viously dene what is called a negligible effect (e.g., 5%, 10%,20%) as is done when classically estimating x% effectiveconcentrations (ECx) using concentrationresponse curves.
In a bioequivalence study, because only two treatments (twodrugs) are compared, only one condence interval should bedetermined, the one on the ratio of the two means obtained withthe two treatments. In an ecotoxicological bioassay, severaltreatments (several concentrations of a contaminant) are com-pared with the control, using multiple comparisons, in thecontext of which determining condence intervals is moredifcult. However, simultaneous condence intervals for rela-tive effects between several treatments and a control wererecently developed [10,11] and provided in packages associatedwith the R statistical environment . The purpose of thepresent study is to illustrate how such packages may be used toanalyze ecotoxicological bioassays and how simultaneous con-dence intervals may be interpreted by analogy to what isperformed in bioequivalence trials, thus naturally lling thegap between NOEC and ECx values.
MATERIALS AND METHODS
In recent years, several methods have been developed tocalculate simultaneous condence intervals for ratios of means,especially ratio-to-control associated with the Dunnett proce-dure, the most common method used to assess the signicanceof differences between the control response and responses at allother concentrations . We chose to calculate simultaneous95% condence intervals for ratios of each mean to the mean ofthe control using the plug-in method provided within the R
Environmental Toxicology and Chemistry, Vol. 30, No. 12, pp. 28882891, 2011# 2011 SETAC
Printed in the USADOI: 10.1002/etc.686
* To whom correspondence may be addressed(email@example.com).
Published online 19 September 2011 in Wiley Online Library(wileyonlinelibrary.com).
packagemratios . This method calculates approximate simul-taneous condence intervals from a multivariate t distribution,which correlation matrix is estimated from the maximum like-lihood estimation of the ratios. The very simpleR code required tocalculate and vertically plot these condence intervals is providedin the Appendix.
To help interpret the results, horizontal lines were added toeach plot, corresponding to ratio-to-control values of 1 (higherline), 0.95, and 0.90, that is, to no effect, 5% effect, and 10%effect respectively. Considering an x% effect and its corre-sponding horizontal line, if the ratio-to-control condenceinterval for one concentration is entirely below or above thisline, we can say that the ECx is, respectively, below or abovethis concentration. When the condence interval overlaps theline, nothing can be concluded. Looking at the condenceintervals for all the tested concentrations, it is thus generallypossible to bound the ECx whatever x, including 0 (EC0corresponding to a zero-effect, often incorrectly replaced bythe NOEC). Bounds obtained in this way do not strictly denecondence intervals, but they may be interpreted as bounds ofuncertainty intervals: they represent what may be inferred onECx using condence intervals on ratios to control.
Uncertainty intervals were thus determined for EC0, EC5,and EC10. For the sake of comparison, EC5 and EC10 were alsoestimated from the t of the three-parameter log-logistic modelor the four-parameter Brain-Cousens model in case of observedhormesis , using the R package drc .
For our purpose, we analyzed four data sets collected duringstandard 21-d Daphnia magna bioassays. Daphnids wereexposed either to Zn (0, 0.074, 0.22, 0.66mg/L) , Cu
(0, 2.5, 5, 10mg/L) , chlordane (insecticide) (0, 0.18,0.73, 1.82, 2.9, 7mg/L) , or Cd (0, 0.37, 0.86, 1.64,4.36mg/L) (H. Delhaye and B. Clement, Universite deLyon, personal communication). Observed data were bodylength measurements (inmm) at the end of the experiment(after 21 d) as shown in Figure 1.
RESULTS AND DISCUSSION
Simultaneous 95% condence intervals (CI) for ratio ofmeans to control mean were reported in Figure 2 for each studiedcontaminant and each tested concentration. Bounds for EC0,EC5, and EC10were then easily deduced from these plots. Let ustake the example of the inference on EC10 value for Zn: lookingat the 95% CI for ratios of mean to control mean for each testedconcentration, for 0.22mg/L it is entirely above the horizontalline corresponding to a ratio of 0.90 (a 10% effect), whereas for0.66mg/L it is entirely below that line. It thus can be inferred thatthe EC10 value lies between these two concentrations. For eachdataset, Table 1 reports bounds for EC0, EC5, and EC10 likewisededuced from 95% simultaneous intervals on ratio-to-control.Table 1 also shows 95% condence intervals on EC5 and EC10as estimated from the tting of the 3-parameter log-logisticmodel or the Brain-Cousens model (for Cu).
For EC0, all of the intervals are left bounded by 0 andright bounded by the LOEC. But with a lower bound at 0,such an interval cannot help regulators to dene acceptableeffect levels. To obtain a more informative interval from aDunnett procedure, one must specify what is considered to be anegligible effect, and so infer on ECx values, with a non-nullvalue of x.
Zinc concentration (mg.L1)
Copper concentration (g.L1)
0 1 10
Chlordane concentration (g.L1)
0 0.1 1
Cadmium concentration (g.L1)
Fig. 1. Daphnidbody lengthmeasurements as a functionof concentration (log scale),with the log-logisticmodelor theBrain-Cousensmodel (forCu)tted todata.
Filling the gap between NOEC/LOEC and ECx concepts Environ. Toxicol. Chem. 30, 2011 2889
The EC5 and EC10 bounds given by tting a doseresponsemodel or dened from simultaneous condence intervals are inagreement, always overlapping. That should be conrmed bycomparing both methods on a greater number of data sets. Incontrast to NOEC-based methods, the method using simulta-neous condence intervals can represent the uncertainty attrib-utable to the number of observations or to the variability ofobservations. For example, with the Cd data set, the low numberof replicates induces a great uncertainty on the EC5 (Figs. 1and 2 and Table 1). Moreover, intervals proposed in the presentstudy do not rely on any model assumption; this may be seen asan advantage over the classical ECx, sometimes criticizedbecause of being model-dependent [2,17]. Nevertheless, thatmodel-dependence may not be such a problem when toxicityendpoints are estimated by interpolation (without extrapolation)
 and, on the four tested data sets, the regression methodgave smaller uncertainty intervals than those we proposed thatare inevitably bounded by concentrations of the experimentaldesign (Table 1). Thus, if the method proposed in the presentstudy appears to be a good alternative to NOEC-based methodfor data designed to be analyzed with the Dunnett procedure, thet of models may be a more efcient way to estimate ECxvalues, but it should be accompanied by an adaptation ofexperimental designs, including more closely spaced treatmentsand fewer replicates .
Following the same objective of preventing the wronginterpretation of the NOEC as a concentration that would causeno toxic effect, some authors also advised that the NOEC shouldbe accompanied by the minimum signicant difference atthe NOEC . Indeed, such a measure indicates how much
Zinc concentration / control (mg.L1)
0.074/0 0.22/0 0.66/0
Copper concentration / control (g.L1)
2.5/0 5/0 10/0 20/0
Chlordane concentration / control (g.L1)
0.18/0 0.73/0 1.82/0 2.9/0 7/0
Cadmium concentration / control (g.L1)
0.37/0 0.86/0 1.64/0 4.36/0
Fig. 2. Simultaneous 95% condence intervals for ratios of body length mean mk for each concentration ck, to the mean m0 of the control (concentration c0,generally 0), with lines indicating no effect (EC0 : ratio 1), 5% effect (EC5 : ratio 0.95), and 10% effect (EC10 : ratio 0.90). On the x label the twoconcentrations ck and c0 are indicated as ck/c0.
Table 1. Bounds for EC0, EC5, and EC10 values deduced from 95% simultaneous condence intervals (CI) on ratio-to-control (Fig. 2) and 95% condenceintervals on EC5 and EC10 values estimated from the tting of the log-logistic or the Brain-Cousens model (for copper) (Fig. 1)
Bounds for EC0 from 95% CI 0; 0.074 0; 10 0; 0.73 0; 0.37Bounds for EC5 from 95% CI 0.074; 0.22 10; 20 0.18; 2.9 0; 4.3695% CI on EC5 log-logistic model 0.11; 0.18 10.6; 12.4 0.41; 1.24 0.74; 1.61Bounds for EC10 from 95% CI 0.22; 0.66 10; 20 1.82; 2.9 1.64; 4.3695% CI on EC10 log-logistic model 0.24; 0.31 13.6; 16.1 1.28; 2.48 2.34; 3.40
EC effective concentration.
2890 Environ. Toxicol. Chem. 30, 2011 M.-L. Delignette-Muller et al.
difference in response should be observed minimally for aresponse to be found signicantly different from that observedin the control. Unfortunately, this measure adds one morestatistical concept, which may not be so simple to understandand to apply for regulators. Hence, the use of simultaneouscondence intervals proposed in the present study appearseasier, because it is directly linked to well-known ECx values.
The method we propose, based on the use of simultaneouscondence intervals on ratio-to-control, provides an easy andproper way to interpret the results of a Dunnett procedure andprevents the common misuse of p values associated with LOECor NOEC calculations. It provides the result as an uncertaintyinterval for whatever ECx values, even EC0 (zero effect con-centration). In fact EC0 may only be upper-bounded by theLOEC value, which gives little useful information when theobjective is to dene an acceptable effect level . Thus, toproperly infer useful information from the commonly usedDunnett procedure, one must previously specify what is calleda negligible effect, and to work on ECx values (with a non-nullvalue of x) as classically done with doseresponse curves.Another interesting way to interpret such data, without deningan x% effect, is the estimation of the no effect concentration as aparameter of a model describing the occurrence of an effectabove a threshold concentration [15,21,22].
AcknowledgementWe thank developers of the two R packages used in thepresentwork,G.DilbaDjira andhis collaborators for themratiospackageandC. Ritz and J. Strebig for the drc package. We also thank R. Manard and hiscollaborators and B. Clement and H. Delhaye for kindly providing us withtheir data.
1. Organisation for Economic Co-operation and Development. 2006.Series onTesting andAssessmentNumber 54. Current approaches in thestatistical analysis of ecotoxicity data: A guidance to application. OECDEnvironment Health and Safety Publications, Paris, France.
2. van der Hoeven N. 1997. How to measure no effect? Part III: Statisticalaspects of NOEC, ECx and NEC estimates. Environmetrics 8:255261.
3. Fox DR. 2008. NECs, NOECs and the ECx. Aust J Ecotoxicol 14:79.4. NewmanMC. 2008. What exactly are you inferring? A closer look at
hypothesis testing. Environ Toxicol Chem 27:10131019.5. WarneMSJ, van DamR. 2008. NOEC and LOEC data should no longer
be generated or used. Aust J Ecotoxicol 14:15.6. Altman DG, Bland JM. 1995. Absence of evidence is not evidence of
absence. Br Med J 311:495.7. ArmstrongJS. 2007.Signicance tests harmprogress in forecasting. Int J
Forecasting 23:321327.8. Gardner MJ, Altman DG. 1986. Condence intervals rather than P
values: Estimation rather than hypothesis testing. Br Med J 292:746750.
9. U.S. Food and Drug Administration. 2001. Guidance for industry:Statistical Approaches to Establishing Bioequivalence and hazardousconcentration in ecotoxicology. Washington, DC.
10. DilbaG,Bretz F,GuiardV,HothornLA. 2004. Simultaneous condenceintervals for ratios with applications to the comparison of severaltreatments with a control. Methods Inf Med 43:465469.
11. HothornT,Bretz F,Westfall P., 2008. Simultaneous inference in generalparametric models. Biometrical J 50:346363.
12. R Development Core Team. 2011. R: A language and environment forstatistical computing. R Foundation for Statistical Computing, Vienna,Austria.
13. Brain P, CousensR. 1989.An equation to describe dose responses wherethere is stimulation of growth at low dose. Weed Res 29:9396.
14. Ritz C, Streibig JC., 2005. BioassayAnalysis usingR. J Stat Softw 12:122.
15. Billoir E, Delignette-Muller ML, Pery AR, Charles S. 2008. A Bayesianapproach to analyzing ecotoxicological data. Environ Sci Technol42:89788984.
16. Manar R, Bessy H, Vasseur P. 2009. Reproductive effects andbioaccumulation of chlordane in Daphnia magna. Environ ToxicolChem 28:21502159.
17. Isnard P, Flammarion P, Roman G, Babut M, Bastien P, Bintein S,Essermeant L, Ferard JF, Gallotti-Schmitt S, Saouter E, Saroli M,Thiebaud H, Tomassone R, Vindimian E. 2001. Statisical analysis ofregulatory ecotoxicity tests. Chemosphere 45:659669.
18. Moore DRJ, Caux PY. 1997. Estimating low toxic effects. EnvironToxicol Chem 16:794801.
19. van der Hoeven N. 2008. Calculation of the minimum signicantdifference at the NOEC using a non-parametric test. Ecotoxicol EnvironSaf 70:6166.
20. van der Hoeven N. 2004. Current issues in statistics and models forecotoxicological risk assessment. Acta Biotheor 52:201217.
21. KooijmanS,Bedaux J., 1996.TheAnalysis of Aquatic ToxicityData. VUUniversity Press, Amsterdam, The Netherlands.
22. Fox DR. 2010. A Bayesian approach for determining the no effectconcentration Ecotoxicol Environ Saf 73:123131.
Beginning users of R can nd information in one of themultiple free manuals provided on the Comprehensive RArchive Network in the sections Manuals or Contributed(www.r-project.org).
First one must load the mratios package using the followingcommand in R: require(mratios)
You may then use the function sci.ratio to calculatecondence intervals, by specifying in a formula (eg,response treatment) the numerical response (response) andthe grouping factor (treatment) coding for various concentra-tions and control, in a data frame the name of which must alsobe given as the second argument of the function.
s < sci:ratio formula response treatment;data nameofthedataframe
By default, two-sided 95% condence intervals are calcu-lated for the Dunnett contrast type, using the rst level of thegrouping variable as the control, but various other arguments ofthe function may be modied, such as the condence level orthe level of the grouping factor corresponding to the control. Acomplete list of sci.ratio arguments is provided in its help leaccessible using the following R command:
After being calculated, condence intervals may be printedand plotted by simply applying the print and plot functions tothe object returned by the sci.ratio function. In the followingcommand using the plot function, the argument CIvert is xedat TRUE so as to plot condence intervals vertically; a vector ofvalues dening the ratios of interest is affected to the argumentrho0, so as to plot corresponding horizontal lines, as in Figure 2.
printsplots; rho0 c1; 0:95; 0:90; CIvert TRUE
Filling the gap between NOEC/LOEC and ECx concepts Environ. Toxicol. Chem. 30, 2011 2891