In psychological and social science research, behaviour is most often assessed by self-report questionnaires using Likert scales (Baumeister et al., 2007; Clark & Watson, 2019; Sullivan & Artino, 2013). It is common practice to create pools of Likert items to form subscales which each represent an aspect of the overall construct that the questionnaire is intended to investigate. Single Likert items (i.e., questions) are not considered to sufficiently capture these aspects (Clark & Watson, 2019; Rickards et al., 2012) and are therefore summarized into subscales by considering either the sum or average of subgroups of Likert items. The development and best practices of constructing questionnaires using Likert-type responses is discussed in the methodological psychology literature (Jebb et al., 2021). Likert (1932) developed the typical 5- or 7-point ordinal scale on which single items are measured, e.g., ranging from “Strongly approve” to “Strongly disapprove”. He suggests to assign numerical values to the answer choices in the same order as they are ranked. However, he does not suggest that these ordinal values must necessarily be translated into an equidistant scale, and states that the same results will be obtained as long as the rank order is preserved. This translation of an ordinal scale into a numerical scale conditional on rank preservation is considered to be legitimate elsewhere (Silan, 2020). So in conclusion, the distances between the numerical values are irrelevant to the analysis (Gaito, 1980) which complies with the ordinal measurement scale of the Likert items where distances between answer choices cannot be measured. Likert (1932) suggests to subsequently take the sum or mean of the transformed values, which he assumes to be normally distributed. There is a long-standing debate about how Likert-type scales should appropriately be analysed but the prevailing opinion due to vast empirical evidence (Carifio & Perla, 2007; Norman, 2010) is that survey scales as opposed to single Likert items may be treated as interval data such that means and standard deviations can be computed, and parametric methods should be applied to them (Carifio & Perla, 2008; Rickards et al., 2012; Sullivan & Artino, 2013). Specific examples for the application of LDA to questionnaire data based on Likert-type scales are Knowles et al. (2000), Kristjansdottir et al. (2018), Veronese and Pepe (2017), and Wang et al. (2016). In all of these studies, the authors computed Fisher discriminant function coefficients (descriptive DA) for the subscales of the considered psychological questionnaires using Likert-type responses and showed the validity of these coefficients, i.e., their discriminative ability, by subsequent linear classification (predictive DA). In particular, Wang et al. (2016) examine a longitudinal data set but restrict their analysis to time point one when applying LDA. Veronese and Pepe (2017) emphasize the need to explore the dynamic relations between their chosen subscales over time and point out their restriction to cross-sectional data in their LDA as a considerable limitation.

Two datasets differing in the number of repeated measurement occasions, as well as two modifications thereof, are used as reference datasets. Each original dataset comprises measurements of four continuous predictor variables which are measured at two time points (CORE-OM dataset) and four time points (CASP-19 dataset), respectively. The binary outcome variable represents the group (y∈{0,1}). Both of these standardized psychological questionnaires consist of Likert-type questions measured on a 5-point and 4-point Likert scale, respectively. According to the developers of these questionnaires, we considered the mean score of multiple Likert items in case of the CORE-OM dataset, and the sum score in case of the CASP-19 dataset, respectively, as the basis for parameter estimation and subsequent data simulation.

We created reference datasets from these data in order to compare the methods’ performance in different (almost) realistic settings, not in order to draw any substantive conclusions about the data themselves. Datasets differ among others in sample sizes, sample size ratios, class overlap, temporal variation, and number of measurement occasions.

The first dataset (Zeldovich, 2018) is a self-report questionnaire of psychological distress abbreviated to CORE-OM (Clinical Outcomes in Routine Evaluation-Outcome Measure) (Barkham et al., 1998). It assesses the progress of psychological or psychotherapeutic treatment using four domains (subjective well-being, problems/symptoms, life functioning, risk/harm) measured on a 5-point Likert scale (0: not at all, 1: only occasionally, 2: sometimes, 3: often, 4: most or all the time). Our dataset uses the binary variable hospitalisation as group variable and is denoted as “Dataset 1” in the following. Non-hospitalised participants represent Group 0 (n0=42) and hospitalised ones Group 1 (n1=142).

The second dataset is a self-report questionnaire of quality of life developed for adults aged 60 and older abbreviated to CASP-19 (Hyde et al., 2003). The dataset on CASP-19 is derived from Waves 2, 3, 4, and 5 of the English Longitudinal Study of Ageing (ELSA) (Banks et al., 2021). The CASP-19 questionnaire comprises four subdomains (Control, Autonomy, Self-realization, Pleasure) measured on a 4-point Likert scale (0: Often, 1: Sometimes, 2: Not often, 3: Never; reversed scale for some items). Loneliness as one of the factors affecting quality of life (Talarska et al., 2018) is chosen as the group variable. For this purpose, the sample was dichotomized at a score value of three determined from two questions related to loneliness (“Old age is a time of loneliness”, “As I get older, I expect to become more lonely”), answered on a 5-point Likert scale (1: Strongly agree, 5: Strongly disagree) by the participants during Wave 2. Persons who feel less lonely represent Group 0 (n0=948) and those who feel more lonely represent Group 1(n1=1682). Since the group differences were nevertheless marginal, we modified these data. All individuals of Group 1 were included in our reference dataset, but only those individuals of Group 0 were included, whose scores of the variables “control” and “self-realization” lay above their respective 0.2 percentiles. The dataset is referred to as “Dataset 2” in the following.

Answers to questions of each subdomain in these questionnaires using Likert-type responses are summarized in a score, where a higher mean score correspond to a higher level of distress (Dataset 1), and a higher sum score indicates a better quality of life (Dataset 2), respectively. Data simulations are based on these scores. Boxplots in Figure 1a and 1b show the scores’ distribution in Reference Data 1 and 2, respectively. They indicate that on average individuals in one group usually obtain higher/lower scores compared to the other group irrespective of the time point and variable, presumably facilitating classification in these datasets. Also, temporal variation in Dataset 2 is rather modest.

Therefore, we considered further scenarios beyond these two original datasets (Dataset 1: CORE-OM, Dataset 2: CASP-19). In addition, to test the methods under different conditions, we provided three modified versions of these datasets (Dataset 3: modified CORE-OM with equal group means collapsed over time points and group means with opposite temporal trends, Dataset 4: modified CASP-19, Time Points 1 & 2 only, with identical means but heterogeneous covariance matrices, Dataset 5: modified CASP-19, Time Points 1 and 2 only, balanced class sizes by random undersampling of Group 1). Dataset 3 was modified by adding a constant specific to each variable to the data of Group 0 such that collapsed means of both groups became equal in size, while maintaining the original boundaries of the measurement intervals. Then we swapped the data of the two time points for Variables 1, 2, and 3 for Group 0, such that means of Group 0 have an upward temporal trend compared to the downward temporal trend of measurements in Group 1. For Dataset 4, only Time Points 1 and 2 are considered. We adjusted group means of Group 0 per time point such that they equal those of Group 1. For Dataset 5, also only Time Points 1 and 2 are considered, and a random subset of the larger Group 1 equalling the sample size of Group 0 was chosen in order to obtain a balanced scenario. The corresponding R code can be found in Graf et al. (2025, code “availability”).

With Dataset 4, the aim was to create data which only differ in their group covariance matrices. Homogeneity of covariance matrices can be tested using the well-known Box’s M test (Box, 1949), but its reliability suffers when the multivariate normality assumption is even only slightly violated (Tiku & Balakrishnan, 1984). For Dataset 4, the p-value of the approximate χ2 test statistic of the Box’s M test is <.001( χ2(36) = 1789.9) but this significant test result may indicate a violation of normality instead of inequality of covariance matrices. Therefore, since the data significantly differ from multivariate normality (Table S4 in Graf et al., 2025), we visually assessed the covariance matrices’ heterogeneity based on the components used for Box’s M test, i.e., log determinants of the pooled and group covariance matrices (Σpooled,Σ0,andΣ1), which equal the product of their respective log eigenvalues. We use plots of log determinants with 95% confidence intervals and plots of log eigenvalues of the covariance matrices as suggested by Friendly and Sigal (2020). From Figure 2 we conclude substantial heterogeneity of the group covariance matrices Σ0 and Σ1 in Dataset 4. For Dataset 4, simulations are based on the estimates of Σ0 and Σ1, whereas for Datasets 1–3, and 5 they are based on the estimate of Σpooled such that the LDA assumption of homogeneous covariance matrices holds. Figure S1 in Graf et al. (2025) shows the plots for inspecting heterogeneity of covariance matrices for the other reference datasets as well. The assumption is not fulfilled in any of the datasets. Boxplots in Figure 1c–1e show the scores’ distribution in Reference Data 3–5.

We chose reference datasets with moderate temporal and cross-sectional correlations. Correlation matrices are shown in Table S1a–S1e of Graf et al. (2025). In this case, analyzing the data separately per time point or focussing on measurements of single variables over multiple time points, respectively, would ignore these correlations and yield less reliable results if, in fact, affiliation to one of the groups is affected by multiple correlated variables and/or time points (e.g., Gnanadesikan & Kettenring, 1984).

For LDA, the unknown parameters μi∈Rdt, i.e., the group-specific mean vectors, and Σ∈Rdt×dt, i.e., the pooled covariance matrix, need to be estimated from the data

Xij1T,⋯,XijtTi∈{0,1}j=1,⋯,ni∈Rn×dt,

where Xijk∈Rd are continuous measurements. Here, i∈{0,1} represents the group label, j∈{1,…,ni} the patient, k∈{1,…,t} the time point, and d the number of variables. The total sample size is denoted by n=n0+n1. The covariance matrix Σ∈Rdt×dt is assumed to be positive definite. The traditional LDA assumes multivariate normality of the data,

Xi∼iidNdt(μi,Σ),

as well as equality of group covariance matrices (homoscedasticity), Σ0=Σ1=Σ. Brobbey (2021) and Brobbey et al. (2022) developed two approaches for robust LDA (when data deviate from multivariate normality) based on the Kronecker product estimate of the covariance matrix Σ that will be described in the Robust Trimmed Likelihood LDA for Multivariate Repeated Measures Data section and in the Generalized Estimation Equations (GEE) Discriminant Analysis for Repeated Measures Data section. Here, we will briefly explain the rationale behind these modified LDA approaches and introduce the general LDA classification rule.

Assuming that Σ is unstructured, all distinct correlations between each pair of the d variables and each combination of the t time points must be estimated. If the dataset is small, the estimate Σ̂ may become singular, i.e., if n≤dt. In order to reduce the complexity of Σ or to estimate Σ more efficiently, a reduced number of parameters can be considered by assuming, for example, a Kronecker product structure Σ=Σt×t⊗Σd×d. Here, Σt×t∈Rt×t comprises the correlations between the t time points and Σd×d∈Rd×d comprises the correlations between the d variables. The number of unknown parameters reduces from (dt(dt+1)/2) for an unstructured covariance matrix to d(d+1)/2+t(t+1)/2 for a Kronecker product covariance matrix (Naik & Rao, 2001). It can be estimated by the flip-flop algorithm, which gives maximum likelihood estimates of Σt×t and Σd×d (Lu & Zimmerman, 2005). The flip-flop algorithm is suitable in case each observation can be separated with respect to two factors, such as the time points and variables in case of multivariate longitudinal data.

The LDA classification rule states that a new observation Xij∈Rdt is assigned to class 0 if Xij−μ0+μ12TΣ−1(μ0−μ1)>logπ1π0

where πi,i∈{0,1}, is the prior probability of class i, μ0 and μ1 the respective group means, and Σ−1 is the inverse covariance matrix (Lix & Sajobi, 2010). In the methods by Brobbey (2021) and Brobbey et al. (2022), Σ−1 is replaced by Σt×t−1⊗Σd×d−1.

The original linear SVM for cross-sectional data and linearly separable classes (Vapnik, 1982) has been modified such that an overlap between the samples of both classes is to some extent allowed (Cortes & Vapnik, 1995). Chen and Bowman (2011) further generalized this SVM classifier such that it becomes applicable to longitudinal data. In their longitudinal SVM algorithm, temporal changes are modeled by considering a linear combination of the observations Xij=xj∈Rdt and a parameter vector β=(1,β1,…,βt−1), which represents the coefficients for each time point k. Then, x~j=xj1+β1xj2+⋯+βt−1xjt, are provided as input to the traditional SVM. Combining the d observations from all t time points in a single vector assumes that the distances between time points are the same. The approach also assumes a fixed number of d observations per time point k (complete data) just as in case of LDA. Although this SVM classifier can also estimate nonlinear decision boundaries depending on the type of kernel matrix that is used, we apply a linear kernel in order to compare its performance to the other linear classifiers and since the absolute values of the weight vector can be interpreted as variable importance in case of a linear kernel matrix.

Although the SVM algorithm does not make any distributional assumptions, the regularization Parameter C needs to be optimized. We use the SSVMP algorithm (Sentelle et al., 2016), a modification of the SVMpath algorithm (Hastie et al., 2004) to find the optimal value of C. The SSVMP algorithm is applicable for unequal class sizes and semidefinite kernel matrices in contrast to the original version by Hastie et al. (2004). The path algorithm finds the optimal value λSVM=1/C with high accuracy, since it considers all possible values of C. At the same time, it is computationally efficient compared to the generally recommended grid search. It has been shown that the choice of C can be critical for the generalizability of the SVM model (Hastie et al., 2004).

The SSVMP algorithm (Sentelle, 2015; Sentelle et al., 2016) optimizes the inverse of the regularization parameter, λSVM=1/C. Starting with a high value of λSVM such that all samples lie within the margin of the SVM, it successively determines a strictly decreasing sequence of λSVM values for which the set of support vectors changes for each λSVM value, and it stops if no more observations are left inside of the margin (linearly separable case) or if the next λSVM value would be zero.

The longitudinal SVM algorithm by Chen and Bowman (2011) requires to specify a maximum number of iterations used for finding the optimal separating hyperplane parameters. In our case, the iterative algorithm for optimization of the Lagrange multipliers α and temporal change parameters β in the longitudinal SVM is repeated until the Euclidean distance between two consecutive estimates of αm becomes less than 1E−08 or the maximum number of 100 iterative steps is reached. A summary of the longitudinal SVM algorithm using the linear soft-margin approach can be found in Supplementary Material S.2. (see Graf et al., 2025)

Our simulation study aims at mimicking reference datasets from psychological applications. See the Reference Datasets sub-section for a detailed description of these datasets. A brief overview of the steps in the simulation study is given in Figure 3. For each scenario, 2000 datasets are simulated. Sample sizes for the training data are chosen identical to the sample sizes of the reference datasets. Sample sizes for the test data for each group are 10 times the number of the respective original group sample size in order to maintain the group size ratio. A larger test sample size can be chosen in simulations since they do not rely on actual data. Variance in the performance estimates may thereby be decreased. Data are simulated from the multivariate normal distribution (as a reference), from the multivariate truncated normal distribution which only takes on values within specified boundaries similar to the sum or mean scores in the reference data, respectively, and from the multivariate lognormally distributed data in order to include an extremely skewed distribution (overview in Table 3). Parameters needed for data simulations are estimated from the reference datasets (i.e., the pooled covariance matrix Σ, or the group covariance matrices Σ0 and Σ1, respectively, group means μ0, and μ1, and the lower and upper boundaries, a and b, of the sum or mean score, respectively). Training data are either not trimmed or trimmed using the MCD and the MVE algorithm, respectively, keeping 90% of the samples, before applying the classification algorithms. In contrast to Brobbey (2021) and Brobbey et al. (2022), we did not use the restrictive assumption of a Kronecker product covariance structure for simulating the data. In contrast to Chen and Bowman (2011), the datasets to which we applied the method are not balanced in sample size. We would like to examine the methods’ performance in more general simulation settings.

Since the SVM algorithm relies on the Euclidean distance to determine the optimal decision boundary, standardization is required as a data-preprocessing step. We standardized the data variable-wise (across time points) before applying the method. Centering and scaling is done using the preProcess function in the R package caret (Kuhn et al., 2024). More specifically, each training dataset is centered, and scaled to unit variance, and the same parameters are then used to standardize the test dataset in the same way (Hsu et al., 2003). Machine-learning algorithms generally require the optimization of hyperparameters. Application of the linear SVM algorithm requires finding the optimal value of the Hyperparameter C which determines the maximum amount of overlap allowed between samples of both classes. We applied the simple SVM path (SSVMP) algorithm by Sentelle et al. (2016) as suggested by Chen and Bowman (2011) in order to determine the optimal regularization Parameter C. It is available as MATLAB code (Sentelle, 2015), which we rewrote in R. Computation of the longitudinal SVM results including the computation of the optimal C could only be done for the two smaller datasets (Dataset 1 and Dataset 3) due to limitations by computational complexity.

The flip-flop algorithm (Lu & Zimmerman, 2005) used by Brobbey (2021) for estimating the Kronecker product structure of the covariance matrix from the training data (for the LDA(ΣKP) algorithm) was iterated until the Frobenius norm of two consecutive Kronecker product covariance matrices became less than or equal to 1E−04, a proposed stopping criterion by Castañeda Garcia and Nossek (2014).

We used the following software for data simulations. We implemented the longitudinal SVM in R using the R package Rcplex (Bravo et al., 2021). We used the implementations of the MVE and MCD algorithm from the R package MASS (Ripley et al., 2022), the joint GEE model as implemented in the R package JGEE (Inan, 2015), and implemented the version of the flip-flop algorithm in R as described in Lu and Zimmerman (2005). For simulation of multivariate normally, lognormally, and truncated normally distributed data, we used the respective functions from the R packages MASS (Ripley et al., 2022), compositions (van den Boogaart et al., 2022), and tmvtnorm (Wilhelm & Manjunath, 2022). For the truncated normal distribution, the rejection method (default) was used.

For computing point estimates of the performance measures including confidence intervals in the reference data, we used the bootstrap approach described in the Nonparametric Bootstrap Approach sub-section. Estimates of predictive performance and the Youden index are shown in Figure 4, those of sensitivity and specificity can be found in Figure S2 (see Graf et al., 2025). The bootstrap estimates and their respective confidence intervals are also shown in Table S3 (see Graf et al., 2025).

Figure 4 shows that the two methods LDA(Σpooled) and LDA(ΣKP) have a very similar performance in all scenarios, and generally perform best. Including Figure S2 in Graf et al. (2025), these two methods tend to have more moderate values of sensitivity and specificity even for highly imbalanced datasets (Datasets 1,2,3). However, similar to LDA(GEE), they are (almost) incapable to accurately predict the correct class of individuals from the minority class when group means are identical and only group covariance matrices differ (Dataset 4). All three methods, LDA(Σpooled), LDA(ΣKP), and LDA(GEE), predominantly predict that individuals belong to the majority class in this scenario, probably because its covariance matrix has a greater weight when computing the inverse of the pooled covariance matrix for the classification rule (Multivariate Repeated Measures LDA sub-section). In comparison, LDA(GEE) and SVM perform worse for unequal class sizes, of which SVM performs worse compared to LDA(GEE), particularly because its specificity (prediction of the minority class) is very low. Comparing the performance for Dataset 1 (same temporal trends of group means) and Dataset 3 (opposite temporal trends of group means), the results of all performance measures considerably improve for LDA(Σpooled) and LDA(ΣKP). For LDA(GEE) there is almost no change (very slight improvement), and overall no difference for the SVM. Results for Dataset 5 show that using balanced instead of the imbalanced data (Dataset 2), increase specificity of all LDA methods but particularly for LDA(GEE), resulting in a higher Youden index. Trimming of the training data does only in some cases improve the performance in the test data. A slight improvement of predictive performance and Youden index at the same time can only be observed in some cases: for LDA(Σpooled) when applied to Dataset 1 (after MVE trimming), Dataset 3 (after MVE and MCD trimming), Dataset 5 (after MCD trimming), for LDA(ΣKP) when applied to Dataset 1 (MVE trimming), Dataset 3 (MVE and MCD trimming), and LDA(GEE) when applied to Dataset 1 (MCD trimming).

For data simulations we assumed homogeneity of covariance matrices (which is a LDA assumption) for data generation based on Datasets 1, 2, 3, and 5, despite heterogeneity in the reference datasets. Figure S1 in Graf et al. (2025) shows plots for comparison of the components of Box’s M test, which is known to be very sensitive to violations of the normality assumption, and results may therefore not be reliable. Log determinants and log eigenvalues of the covariance matrices differ from each other suggesting heterogeneity of covariances in the reference data. Only for Dataset 4 we assumed heterogeneous covariance matrices for data generation in order to compare the methods’ performance under violation of this assumption when group means are identical at the same time.

The second LDA assumption is multivariate normality of the data. Table S4 in Graf et al. (2025) shows that lognormally distributed multivariate data differ most from multivariate normality according to the Mardia measure of multivariate skewness (highest absolute number of significant test results). Truncated normally distributed data differ more significantly from multivariate normality for larger sample sizes and/or a higher number of measurement occasions. Especially for Datasets 2 and 4, respectively, trimming the data using the MCD algorithm notably decreases deviation from multivariate normality in truncated normally distributed data, which is also true for datasets 1 and 3, respectively, when the MCD algorithm is applied to the lognormally distributed data. This effect is weaker for the MVE algorithm. This shows at least that the MCD algorithm, which has been found to be more suitable for outlier detection compared to the MVE algorithm (Rousseeuw & Driessen, 1999), may be useful in case outliers or non-normality is assumed to bias parameter estimates. On the other hand, the optimal trimming value has to be chosen in order to not remove valuable observations from the data. There currently are no general guidelines.

Table 4 shows the computational times per algorithm, averaged over scenarios using different data distributions and trimming approaches. The method LDA(Σpooled) has the advantage of low computational times. Especially for LDA(ΣKP) computational time hugely increases with larger sample size and/or higher number of measurement occasions. In comparison, computational time of LDA(GEE) seems to be less affected by larger sample sizes but rather higher dimensionality (number of time points and variables). Computation of SVM results are most time-consuming, and the algorithm does not always converge after 100 iterations (Table S6 in Graf et al., 2025).

Figures 5 and 6 show the estimates’ distribution of predictive accuracy and the Youden index in the simulated data, respectively. Plots for sensitivity and specificity are shown in Figures S 3 and S 4, respectively. Mean (standard error) of the performance measures are also shown in Tables S5a–Se (see Graf et al., 2025) for Datasets 1–5. A first finding from Figures 5 and 6 is that deviation from normality (in the multivariate lognormally distributed data) in some cases increases (Dataset 1), decreases (Dataset 2 and 5) the algorithms’ predictive performance and Youden index, and in some cases does not have a considerable effect (Dataset 3 and 4). It seems that for the scenarios with smaller sample sizes (n0=42,n1=142), no negative effect could be determined, whereas for the scenarios with much larger sample sizes (n0=948,n1=1682 and n0=n1=948) there is a clear decrease in predictive accuracy and Youden index. The effect is approximately the same for all three repeated-measures LDA methods. A second finding is that predictive accuracy and Youden index for the SVM are visibly worse compared to the LDA methods for these imbalanced sample sizes. It has a sensitivity close to 1, but specificity close to 0, and thus mostly predicts the majority class.

With respect to predictive accuracy, LDA(Σpooled) without prior trimming usually performs best. Only for Dataset 5 (balanced class sizes) LDA(GEE) with prior trimming (MCD algorithm) has a marginally better predictive performance in the lognormally distributed data. Values of both measures, predictive performance and Youden index, of LDA(GEE) are only equal to the other two LDA methods for Dataset 5 (equal sample sizes), Dataset 4 (where all methods perform poorly) and lognormally distributed data simulated based on Dataset 2 (where all methods perform poorly). For Dataset 1 (unbalanced classes, same temporal trends of group means), the Youden index of LDA(GEE) is higher than the values for LDA(Σpooled) and LDA(ΣKP) for multivariate normally and truncated normally distributed data, especially when no trimming is applied to the training data. The boxes only slightly overlap or do not overlap at all. The reason is its higher specificity (prediction of the minority class), but its sensitivity is comparably lower. For the lognormally distributed data generated based on Dataset 1, the Youden index of LDA(ΣKP), especially without prior trimming, is higher compared to the other methods, which is also due to higher specificity.

It is not clear in which situations among the presented simulation scenarios trimming for outlier removal may help, but there is no scenario where we explicitly simulated outliers. Both, predictive accuracy and the Youden index, somewhat increase (from a rather low performance level) for all three LDA methods in the lognormally distributed data for Dataset 5 when trimming in the training data is done.

Generally, in these simulations the traditional LDA(Σpooled) performs best or reasonably well with respect to predictive performance and Youden index, irrespective of smaller or larger sample size, differing group size ratios, number of measurement occasions, similar or opposite temporal trends in group means. None of the LDA methods works well for identical group means but heterogeneous covariance matrices, where they predominantly assign new observations to the majority class, and the Youden index is close to zero. The same is the case for multivariate lognormally distributed data when sample sizes are large, i.e., for an extremely evident violation of multivariate normality corresponding to extremely high values of the Mardia measure of multivariate skewness test statistic (approximately above 100).

We did not explicitly generate outliers from a different distribution than the actual data, but there may have been some random outliers. In this case, trimming for outlier removal had no effect except a minor effect on the Youden index for all LDA methods in the scenario with balanced group sizes and same temporal trends per group when data were generated from lognormally distributed data. In this case, the LDA methods still did not perform reasonably well. Multivariate trimming in the training data can be tried as a sensitivity analysis if the presence of outliers is suspected. Especially the MCD algorithm has already been recommended in the literature.

In our simulations no Kronecker product covariance matrices and group means are assumed in the reference data. We used unstructured estimates of the pooled covariance matrix and group means. In our simulations, there is only an advantage of the alternative LDA(ΣKP) and LDA(GEE) with respect to the Youden index for data with imbalanced class sizes and comparably smaller (but not small) sample sizes. The advantage of these methods, even if no underlying Kronecker product structure of the parameters can be assumed, may become more evident for smaller sample sizes. They may provide more exact estimates due to their parsimonious number of values that have to be estimated.

Application of repeated-measures techniques should be preferred in order to incorporate the additional information about temporal trends and in order to obtain more reliable results by including data of multiple time points in the analysis provided that moderate correlations between data of different variables and times points exist. Multicollinearity among time points and/or variables would require removal of respective time points or variables, respectively. In case of independence between time points/variables, univariate techniques can be used. According to the psychometric literature, multivariate data are very common. An example are the widely applied questionnaires using Likert-type responses where multiple correlated aspects related to an overall topic are measured. In order to assess the usefulness of different sets of variables for distinguishing two classes of individuals, LDA can be applied for class prediction and its performance for different sets of variables can subsequently be compared to determine the most relevant variables. Usually, for LDA applied to cross-sectional data, Fisher discriminant function coefficients (Fisher, 1936) are computed in order to assess relative variable importance within a particular set. The method can in principle also be applied to repeated measures data. It does not assume multivariate normality although it requires homogeneity of covariance matrices.