Informative Hypothesis Testing in the EffectLiteR Framework: A Tutorial

ANOVA

ANOVA is one of the most popular statistical techniques in the social and behavioral sciences. It is a framework or collection of methods based on a linear regression model, where at least one predictor is categorical in nature. Usually, this categorical predictor describes the different conditions of an experiment, or the different (treatment) groups in an intervention study. The ANOVA framework (Edwards, 1993) includes one-way and multi-way ANOVA, univariate and multivariate (M)ANOVA, ANOVA using within-subjects and/or between-subjects factors, and AN(C)OVA where covariates are included in the model.

The problem with ANOVA is that we only obtain regression coefficients as well as main and interaction effects, which are often difficult to interpret. Especially the interpretation of a main effect in the presence of an interaction effect is far from trivial. This is because different sum of squares (SS) can be used in ANOVA for hypothesis testing. Depending on the SS, the main effect is defined in a different way and thus, a different null hypothesis is tested (for an overview, see, e.g., Fox, 2016; Graefe et al., 2022; Maxwell et al., 2018). There are several types of SS, namely Type I, Type II and Type III.² To understand the main differences between these different types, consider a model of the form $Y\sim A+B+A:B$. An ANOVA table based on Type I SS will contain the following three model comparisons: $Y\sim A$ versus $Y\sim 1$,³ $Y\sim A+B$ versus $Y\sim A$, and $Y\sim A+B+A:B$ versus $Y\sim A+B$. In other words, Type I SS corresponds to an incremental procedure, where single terms are added to the model one by one and the model is then compared to the previous model without the new term. For Type II, the ANOVA table will contain the following model comparisons: $Y\sim A+B$ versus $Y\sim B$, $Y\sim A+B$ versus $Y\sim A$, and $Y\sim A+B+A:B$ versus $Y\sim A+B$. The main characteristic of the Type II procedure is the principle of marginality: If a term is removed from the model, then all higher-order terms involving this term will be removed too. Finally, the Type III procedure leads to the following set of model comparisons: $Y\sim A+B+A:B$ versus $Y\sim B+A:B$, $Y\sim A+B+A:B$ versus $Y\sim A+A:B$, and $Y\sim A+B+A:B$ versus $Y\sim A+B$. It is always the full model versus a model where a single term is deleted.

Type III SS are used per default in many popular software programs like SPSS (IBM Corp, 2020). Thus, researchers will typically use Type III SS without further deliberation, even though there is a great controversy in the literature about when to use which SS (e.g., Hector et al., 2010; Herr & Gaebelein, 1978; Macnaughton, 1998). Graefe et al. (2022) conducted simulation studies considering the different types of SS in balanced, proportional and non-orthogonal designs. They found that in balanced designs, using either one of the three SS yields main effects that can be interpreted unambiguously. However, in proportional designs, this is only true when using Type I and II SS. In case of Type III SS, the main effect is biased if there are interactions. Finally, in non-orthogonal designs, the main effect is always biased when using Type I SS. And when there are interactions, Types II and III also yield biased main effects. Nevertheless, for the sake of illustration, we will use ANOVA to analyse our simulated dataset.

Using the centered version of the pre-test variable Z, the model for our running example is fitted as follows in R:

# this R code can be found under 01Anova.R on the OSF project site

# read in data
Data <- read.csv("runningExampleData.csv")

# center pre-test for better interpretation of regression coefficients
Data$pretest.cent <- Data$pretest - mean(Data$pretest)
# treat group and treatexp as categorical variables
Data$group <- factor(Data$group)
Data$treatexp <- factor(Data$treatexp)

# load packages
library(car) # for Anova () function

# fit regression model with all 2-way interaction effects
lmod.treat <- lm(posttest ~ (group + treatexp + pretest.cent)^2, data = Data)
summary(lmod.treat)

# Type I ANOVA table
anova(lmod.treat)

# Type II ANOVA table
Anova(lmod.treat, type = 2)

# Type III ANOVA table
# attention: we must use an orthogonal or a sum-to-zero coding scheme
options(contrasts = c("contr.sum", "contr.poly"))
lmod.sum <- lm(posttest ~ (group + treatexp + pretest.cent)^2, data = Data)
summary(lmod.sum)
Anova(lmod.sum, type = 3)

The anova( ) function is a function in base R, whereas the Anova( ) function belongs to the car package (Fox et al., 2022). The former uses Type I SS, whereas the latter can handle Type II and III SS. Furthermore, R uses treatment coding per default, which has to be changed to a zero-to-sum coding scheme (for example sum coding) when using Type III SS. This prevents main and interaction effects from overlapping. For more information about coding schemes, see, for example, Cohen et al. (2003) and Hardy (2003). Note, however, that in some statistical packages (for example SAS), coding schemes are automatically taken care of. Table 2 shows the ANOVA results when using the three different SS. In Table 3, the results of the linear model using treatment coding can be seen. Appendix Table A1 shows the results of the linear model when using sum coding.

Table 2 shows that the results differ depending on which SS was used. Notably, the results concerning the interaction terms are the same when using Type II and III SS. Generally, the results of the highest order terms (in our case the two-way interactions) are identical between Type II and III SS, but the results of the lower order terms differ. Furthermore, the result of the highest order term when using Type I SS (in our case the treatexp:pretest interaction) corresponds to the result of this term when using Type II and III SS. This term is shown in the last line before the residuals in the results tables and is the only term that has identical results between all three types of SS. Lastly, note that when using Type III SS, the $p$-values of the terms with $df=1$ correspond to the $p$-values of these terms in the linear model when using sum coding (see Table 2 and Appendix Table A1). This is because $t^2=F$ if $df=1$.

Using treatment contrasts and assuming Z is mean centered, we can interpret the regression coefficients in Table 3. Recall that $X=0$ is the control group. The intercept ${\beta }_0$ corresponds to the mean of Y in the control group given $K=0$ and $Z=0$ (see Appendix B for more details), while ${\beta }_1$ denotes the difference between the means of Y in the groups $X=1$ and the control group, given $K=0$ and $Z=0$:

Similarly, ${\beta }_2$ denotes the difference between the means of Y in the groups $X=2$ and the control group, given $K=0$ and $Z=0$:

The difference between the means of Y of $K=0$ and $K=1$ (given the control group $X=0$ and $Z=0$) corresponds to ${\beta }_3$:

And ${\beta }_4$ denotes the expected change in Y for a unit change in Z in the control group if $K=0$:

The change in the effect of $X=1$ versus $X=0$ (${\beta }_1$) between $K=1$ and $K=0$ while keeping Z constant is denoted by ${\beta }_5$. Similarly, ${\beta }_6$ describes the change in the effect of $X=2$ versus $X=0$ (${\beta }_2$) between $K=1$ and $K=0$ while keeping Z constant. The change in the effect of $X=1$ versus $X=0$ (${\beta }_1$) for a unit change in Z when $K=0$ is represented by ${\beta }_7$. Similarly, ${\beta }_8$ denotes the change in the effect of $X=2$ versus $X=0$ (${\beta }_2$) for a unit change in Z when $K=0$. Finally, ${\beta }_9$ describes the change in the effect of $K=1$ versus $K=0$ for a unit change in Z when $X=0$.

Remember that in our example, the researcher assumed an ordering of adjusted means: ${\mathrm{\mu }}_2^{adj}>{\mathrm{\mu }}_1^{adj}>{\mathrm{\mu }}_0^{adj}$. After obtaining a significant main effect of the group variable, the researcher will usually use contrasts to compare the means in depth. Note that in the NHST setting, if no hypothesis about the adjusted means has been specified right from the start, this is called post-hoc testing and should only be used in a descriptive or an exploratory manner. Furthermore, we have to control for familywise error rates (Keselman et al., 2011), which can be done by the emmeans package (Lenth et al., 2022). Concerning our simulated data example, we specify the contrasts in R as follows:

# this R code can be found under 02emmeans.R on the OSF project site
library(emmeans)

# post-hoc tests / contrasts
emmeans(lmod.treat, "group")
emmeans(lmod.treat, "group", weights = "proportional")

emmeans(lmod.treat, "group", contr = "trt.vs.ctrl")
emmeans(lmod.treat, "group", contr = "trt.vs.ctrl", weights = "proportional")

emmeans(lmod.treat, "group", contr = "eff")
emmeans(lmod.treat, "group", contr = "eff", weights = "proportional")

The contrasts are based on the marginal means of Y, which are averaged over the levels of K at the mean of Z. Table 4 shows the marginal means of Y when using equal and when using proportional weights. Using proportional weights implies that the marginal means of Y are averaged over the marginal distribution of K at the mean of Z. This leads to slightly different results in our example compared to using equal weights.⁴

The first set of contrasts (see Table 5) compares the treatment groups with the control group, where the estimates of the differences in marginal means of Y in the levels of X are computed. The second set of contrasts (see Table 6) gives us effect contrasts, where the marginal means of Y are compared with the equally weighed cell means of Y for $Z=0$. Both contrasts have also been computed using proportional weights.

Considering the results, it would seem that the expectations of the researcher are satisfied in the data. That is, the marginal mean of group $X=2$ is higher than the marginal mean of group $X=1$ and the marginal mean of group $X=1$ is higher than the marginal mean of group $X=0$ (see Table 4). Therefore, it seems that there is a greater improvement in the depression post-test Y for CBT with the new drug compared to CBT with the old drug and that there is a greater improvement for CBT with the old drug compared to CBT only. The contrasts agree with this observation. Here, the treatment contrast between group $X=2$ and group $X=0$ is larger than the treatment contrast between group $X=1$ and $X=0$ (see Table 5). Furthermore, the effect contrast is largest for group $X=0$ and smallest for group $X=1$ (see Table 6).

EffectLiteR

Given the different choices for SS when using ANOVA, it becomes clear that main effects are not defined precisely and unambiguously when using ANOVA. However, they are defined precisely and unambiguously in the causal inference literature (see, e.g., Angrist et al., 1974; Neyman, 1990; Pearl, 2009; Rubin, 1974, 2005; Steyer et al., 2000). Common types of effects are the so-called average effects (for example, the effect of a treatment averaged over treatment experience), conditional effects (for example, the effect of a treatment for those without treatment experience only), effects on the treated, effects on the untreated, and so forth. Unfortunately, although the mathematical definition of these effects is well understood, they can be quite complicated and tedious to compute.

EffectLiteR is a framework and an R package that is built upon the clear definitions of effects in the causal inference literature (Imbens & Rubin, 2015; Steyer et al., 2014). Researchers can use EffectLiteR to estimate various effects of interest as well as their standard errors, when the treatment variable is categorical and the outcome variable is continuous. In addition, Wald or F-tests are used to test different hypotheses, for example that (all) average effects are equal to zero, or that (all) conditional effects are equal to zero in the population. For an in-depth introduction into EffectLiteR, see Mayer et al. (2016). In contrast to ANOVA, EffectLiteR directly provides the kind of effects that most applied researchers are interested in.

An average effect is defined as the unconditional expectation of the difference between expected outcomes under treatment and under control. It corresponds to the average causal effect if there are no further unobserved confounding variables and the regression of the post-test Y on the pre-test Z, given a combination of treatment experience K and treatment group X, is in fact linear. Furthermore, an average effect consists of the difference of adjusted means. Considering our running example and groups $X=1$ and $X=0$, this is (Mayer et al., 2016):

After obtaining the descriptive statistics, the researcher can first fit the EffectLiteR model to compute the adjusted means and the effects of interest, but disregard the test statistics and $p$-values. For our running example, this can be done as follows in R:

# this R code can be found under 03EffectLite.R on the OSF project site
	
library(EffectLiteR)

elrmod <- effectLite(
  y = "posttest",
  x = "group",
  k = "treatexp",
  z = "pretest.cent",
  interactions = "2-way",
  method = "lm",
  data = Data
)

print(elrmod)

The full output is shown in Appendix C. The average effects and adjusted means can be found on lines 91–103 and are depicted in Table 7. Note that the average effects correspond to the differences in adjusted means and the subscripts refer to the groups which are compared:

Furthermore, the adjusted means computed by EffectLiteR are very close, but not identical, to the marginal means with proportional weights computed by emmeans. This is because slightly different computations are used.⁵

While ANOVA tests whether main effects are significantly different from zero, EffectLiteR tests whether average effects are significantly different from zero. The number of average effects depends on the number of levels of the group variable. In our example, there is one control and two treatment groups, resulting in two average effects. In short, the EffectLiteR approach is more suitable than the ANOVA approach for testing research questions that focus on average effects. There are just a few rare cases, such as balanced designs, where the results are identical. For further applications of the EffectLiteR approach, see, for example, Flunger et al. (2019), Mayer et al. (2020), Mueller et al. (2015), Rek et al. (2022), and Sadovich (2020).

Type A and Type B Hypotheses

In IHT, a distinction is often made between two types of hypotheses, which are called Type A and Type B hypotheses. The null and alternative hypotheses in our example are Type A hypotheses, which are usually of main interest. When testing Type A hypotheses, $H_{0A}$ (see Equation 14) states that all restrictions are equality restrictions, whereas the alternative hypothesis $H_{1A}$ (see Equation 15) states that at least one inequality restriction is strictly true. Here, the researcher would typically like to obtain a significant result, as this indicates that at least some of the constraints are not equality constraints and thus must be inequality constraints. In contrast, the Type B null hypothesis $H_{0B}$ states that all inequality restrictions hold, whereas the Type B alternative hypothesis $H_{1B}$ states that at least one inequality restriction is violated. When testing Type B hypotheses, the researcher would typically like to obtain a non significant result, because that would imply that we cannot reject the null hypothesis (and thus the expectations of the researcher) based on the data.

If the researcher observes that the expected constraints are satisfied in the data, testing Type A hypotheses suffices. However, if one or more of the assumed constraints are violated in the data, but only to a very small extent, which might be due to sampling variability, the researcher should conduct a Type B hypothesis test before conducting a Type A hypothesis test and correct for multiple testing. We recommend to pre-register this approach.

In our example, Type B hypotheses should be tested if at least one of the two constraints, either $Adj{M}_2>Adj{M}_1$ or $Adj{M}_1>Adj{M}_0$, is violated to a small extent in the data. For example, instead of obtaining the estimates that satisfy the constraints (${\widehat{AdjM}}_0=0.044$, ${\widehat{AdjM}}_1=0.475$ and ${\widehat{AdjM}}_2=0.643$, see Table 7), suppose we would obtain ${\widehat{AdjM}}_2=0.470$, meaning that the constraint $Adj{M}_2>Adj{M}_1$ would be violated to a very small extent. In that case, the researcher should start with testing the Type B hypotheses:

In case the detected violation of a constraint is small, hypothesis test Type B might still be non significant, in which case the researcher can proceed to test hypothesis test Type A. If hypothesis test Type B is significant, then it is clear that the data is contradicting the hypothesis. Therefore, there is no need for testing hypothesis test Type A.

The following syntax illustrates how we can test this Type B hypothesis for our running example:

# this R Code can be found under 04IHT.R on the OSF project site

effectLite_iht(
  object = elrmod,
  constraints = "adjmean2 < adjmean1; adjmean1 < adjmean0"
)
# $test.stat
# [1] "Fbar"
# $Wald.info
# [1] 4.309257e-12
# $pvalue
# [1] 0.6304712

We formulated the Type B hypothesis in terms of the adjusted means, but we could as well have used the averaged effects. Unsurprisingly, when testing $H_{0B}$ against $H_{1B}$, we obtain $Wal{d}_{info}<0.001,p=.630$. We are unable to reject the null hypothesis, and this allows us to proceed and test the Type A hypothesis. Lastly, note that testing a Type B hypothesis before testing a Type A hypothesis rarely seems to alter the conclusions, as can be seen from the simulations conducted by Kuiper et al. (2015).

Further Types of Informative Hypotheses

It is also possible to formulate informative hypotheses using other types of constraints (see e.g. Hoijtink, 2012). For example, effect sizes can be incorporated as in:

Then, Equation 18 corresponds to:

From a substantive point of view, this means that the researcher assumes that the difference between the conditional effect of receiving CBT together with the new drug ($X=2$), given $K=0,Z=0$, and the conditional effect of receiving CBT together with the old drug ($X=1$), given $K=0,Z=0$, is greater than $d$ standard deviations. This may give some indication about the relevance of the difference between the two effects.

“About equality” constraints can be used to test informative hypotheses such as:

Finally, range constraints are a generalization of “about equality” constraints. They can be used to test informative hypotheses like:

Comparison With Equivalence Testing

Equivalence testing (Schuirmann, 1987; Seaman & Serlin, 1998; Wellek, 2010) is a special case of IHT. This is because hypotheses in equivalence testing are formulated using effect sizes, which is also an option in IHT. More specifically, hypotheses in equivalence testing are based on “smallest effect sizes of interest” (SESOIs), which are used to define a range of effect sizes that are of practical interest to the researcher. Equivalence testing became popular in reaction to the replication crisis and is often used in replication studies (see, e.g., Anderson & Maxwell, 2016; Lakens, 2017; Simonsohn, 2015). Here, researchers aim to show that an observed effect is small enough to conclude that its replication was unsuccessful. Generally, equivalence testing can be conducted within the framework of IHT, but IHT allows for a broader range of hypotheses that can be tested.

Regarding our running example, one may apply equivalence testing as follows. Let us assume that our running example is a replication study. We are interested to show that the difference between the raw means of Y in the groups $X=1$ and $X=2$ is small enough to conclude that it is not of practical relevance. In other words, we want to test whether the raw means of Y in the groups $X=1$ and $X=2$ are equivalent (hence the term equivalence testing).⁶ Furthermore, let us assume that the original study had the same sample size as our running example.

For determining a SESOI, we can use one of multiple approaches (see e.g., Lakens et al., 2018). For a discussion on when to use which approach, see Baguley (2009). Here, we will use the popular small telescopes approach (Simonsohn, 2015). It defines the SESOI as the effect size that would give a certain power (say $33\mathrm{\%}$) to the original study. Thus, it indicates the extent to which the replication results are consistent with an effect size large enough to have been detected in the original study (Simonsohn, 2015). We use the value of $33\mathrm{\%}$ power, which is typical for the approach. In our running example, this leads to equivalence bounds of $-0.115$ and $0.115$.

We can then use the TOST (two one-sided tests) procedure (Goertzen & Cribbie, 2010; Lakens et al., 2018; Meyners, 2012; Quertemont, 2011; Rogers et al., 1993). It is implemented in the TOSTER package (Lakens & Caldwell, 2022). The procedure tests the effect estimate, in our case the difference between the raw means, against values at least as extreme as the lower and the upper equivalence bounds. The computations are implemented in R as follows:

# this R code can be found under 05TOST.R on the OSF project site
library(TOSTER)
library(pwr)

# frequency table for group
table(Data$group)
n1 <- 200 # sample size group ’1’
n2 <- 500 # sample size group ’2’

# determining the equivalence bounds via the small telescopes approach
d.33 <- (pwr.t.test(
  n = (n1 + n2) / 2,
  d = NULL,
  sig.level = 0.05,
  power = 0.33,
  type = "two.sample",
  alternative = "two.sided"
))$d
# d.33 = 0.1150074

m1 <- mean(Data$posttest[Data$group == 1])
# 0.438377
m2 <- sd(Data$posttest[Data$group == 1])
# 1.017934
sd1 <- mean(Data$posttest[Data$group == 2])
# 0.7120272
sd2 <- sd(Data$posttest[Data$group == 2])
# 1.047107
m2 - m1
# 0.5795566

# using the raw means and sd
tsum_TOST(
  m1 = m1, m2 = m2, sd1 = sd1, sd2 = sd2,
  n1 = n1, n2 = n2, eqb = d.33, alpha = 0.05, var.equal = FALSE
)
# partial output:
# TOST Results
#                    t	   df     p.value
# t-test        -8.429    533.1     < 0.001
# TOST Lower    -6.756    533.1           1
# TOST Upper   -10.101    533.1     < 0.001

The difference between the raw means is (about) $0.58$ and the equivalence bounds are set to ${\mathrm{\Delta }}_L=-0.115$ and ${\mathrm{\Delta }}_U=0.115$. Using the TOST procedure includes two Welch t-tests. The test against the upper equivalence bound tests $H_0:0.58-{\mathrm{\Delta }}_U=0$ against $H_1:0.58-{\mathrm{\Delta }}_U\ne 0$ and the test against the lower equivalence bound tests $H_0:0.58-{\mathrm{\Delta }}_L=0$ against $H_1:0.58-{\mathrm{\Delta }}_L\ne 0$.

In our example, the test against the upper equivalence bound is significant, $t(533.1)=-10.101,p<.001$, whereas the test against the lower equivalence bound is non significant, $t(533.1)=-6.756,p=1.0$. Since the conclusion of equivalence can only be drawn if both tests are significant, we cannot reject the presence of a difference in raw means between the groups $X=1$ and $X=2$. The classical test from NHST is significant, $t(533.1)=-8.429,p<.001$, indicating that the two groups $X=1$ and $X=2$ statistically differ with respect to their raw means.

In the following sections, we present two empirical data examples that serve to further demonstrate IHT in the EffectLiteR framework. The first example is in the context of linear regression and the second example is in the context of the generalized linear model.