We examine two ML estimators to find the item parameters that maximize the likelihood of the joint empirical distributions of responses and RTs. Estimator ML1 is based on RT categories and fits the deciles of simulated RT distributions from the parameter proposals to the deciles of the empirical RT distribution. ML2 is based on continuous RTs and fits the log kernel density (Jones et al., 2018) of simulated conditional RT distributions for correct responses, incorrect responses and zero-time responses to the empirical conditional RT distributions. While ML1 loses information due to the categorization, the approximation of decile distributions is more stable than the density approximation in ML2. Inference is challenging for both estimators as the likelihood is not calculated analytically. Also, for both estimators, the approximation is contaminated by simulation error.

Even simulation-based approaches can be computationally intensive with repeated costs for simulating data and approximating the likelihood. Neural network approaches are an alternative that have one-off costs to train a parameter estimation network on extensive simulated data and are then fast to output parameter estimates from data input.

The basic idea of a parameter estimation network is to train the connection of given data, i.e. item responses and RTs to point estimates of the six model parameters (Lenzi et al., 2023). During training, the network is exposed to simulated data along with the randomly drawn parameters that generated it. This process aims to enable the network to accurately predict parameter values for any new data that follows the same structure. The estimation network we examine (NN) is based on a network architecture that contains convolutional layers with local connectivity which allows to uphold the data structure in the hidden layers. That is, the response and RT of each trial from the input data is connected to a common node in the first hidden layer. NN contains three convoluted layers, followed by a global average pooling layer and two fully connected layers. Input is generated in batches that contain 1000 pairs of 6 parameters and 1000 simulated trials with responses and RTs. The network is trained on parameter ranges that are derived from the parameter values used in the simulation study (see the subsection on the simulation study’s design), taking the range of each parameter expanded by a length of 1 to each side, with 0 being the minimal value for the parameters s_A , s_B , and s_C. For the training, the parameters are randomly drawn from uniform distributions with these ranges:

The advantage of a network-based approach lies in the rapid estimation once the network is trained. Since simulation-based training is not dependent on empirical data, its one-off costs amortize across the application to any dataset. Inference is possible using a bootstrap approach.

Ideally, the hidden convolutional layers of an estimation network learn sufficient statistics for the estimation. However, since it operates as a black box, it remains uncertain whether this occurs and there is no control over the network’s performance for all possible parameter combinations.

The third group of estimators under examination are Bayesian estimators. The advantage of a Bayesian approach is the availability of credible intervals for parameter inference. Bayesian estimators also accommodate for the variability that is introduced by the simulation.

For all three Bayesian estimators, we employ a differential evolution Markov chain Monte Carlo algorithm (DE-MCMC) (ter Braak, 2006; Turner et al., 2013) that was shown to be an efficient sampler for fitting choice-RT models whose parameters are highly correlated (Turner et al., 2013). The datasets are fitted in 9 chains, each with 20,000 iterations, including a burn-in period of 5,000 iterations. Each of the three estimators uses a unique simulation-based method to approximate the likelihood function, that is implemented in the sampler. As uninformative priors, we chose uniform distributions in parameter ranges that are identical to the ones used to train the network-based estimator NN, shown above.

The first Bayesian estimator we examine is based on one of the neural network approaches to approximate the likelihood function as described by Fengler et al. (2021) (BN). For this, a fully connected network is trained to predict the likelihood value for given parameters, RT and response type (correct or incorrect).

For the second estimator, the likelihood is approximated with a Beta-Weibull distribution whose parameters are predicted by a neural network that has been trained on simulated data (BW).

The third estimator is based on a histogram approach to simulated data to approximate the likelihood function (BH). In this approach, the simulated RTs are split for each response type into 20 item-specific ventiles.

While the examined estimation approaches avoid the computationally expensive analytical likelihood, they are based on simulations that can become computationally costly. The neural network approach has the advantage of becoming cost-efficient after large one-off costs to train the network. The maximum likelihood approach is in general faster than the Bayesian approach but with variable costs depending on convergence. Bayesian approaches are more costly due to the MCMC sampler, but have the advantage of offering better parameter inference with credible intervals.

The convergence of the Bayesian estimators is assessed by examining the R^ values, considering values smaller than 1.1 to be satisfactory (Gelman & Shirley, 2011). To compare the performance, we assess convergence rates: the proportion of parameter-level R^ values smaller than 1.1. Convergence on the parameter level was worst for estimator BN, with an average convergence rate of .64 across all items and parameters, with wide divergence from .42 for Item 2 to .94 for Item 4. Considerably better convergence is found for estimator BW, with an overall average of .95, ranging from .92 (Item 1) to 1.00 (Item 4). The best convergence is found for estimator BH, with an overall average of .99, with a minimum rate of .96 for parameters of Item 5.

A similar result can be found regarding model-level convergence, where BN converges for all parameters in only 52 percent of all 500 item fits (100 datasets with 5 items each), with better rates of .86 and .95 for BW and BH, respectively.

The results of the parameter recovery can be found in Tables 2 and 3. In Table 2, the means and standard deviations of point estimates across all 100 samples are reported with the true values. The means of posterior distributions are used as point estimates for the Bayesian estimators. Table 3 shows the coverage frequency of 0.95 credible intervals for the three Bayesian estimators.

As a first data set to illustrate the use of the DBA model, we chose the 10 items from the Amsterdam Chess Test (ACT; van der Maas & Wagenmakers, 2005) that measure endgame skill in test form "Choose a move A". The participants were chess players that took part in a tournament where the data were collected. Subjects were required to find the best move in a given chess position within a time limit of 30s. The response format was essentially open. Responses were scored as correct or incorrect and RTs were recorded in seconds with a precision of 1/60 of a second. The dataset is available in the LNIRT R package (Fox et al., 2021). Our analysis code, estimation results and diagnostic plots are available at Much and Ranger (2026).

From the original data set with 259 participants, 22 participants with a missing ELO rating were excluded. Responses and RTs to the endgame items were missing from 2 additional subjects. The resulting data set included responses and RTs from N = 235 participants.

We used the Bayesian estimator based on the histogram approach (BH) to fit the DBA model to the data. Starting values, as well as the true parameter values for the simulation study¹1

Estimates for ACT Items 2, 4, 5, 10, and 6 were used as Items 1 to 5 in the simulation study.

Convergence was satisfactory for most items, with exception of Item 7 and Item 6 with R^ > 1.10 for parameter s_C in both items. Applying a stricter convergence criterion by Vehtari et al. (2021), for only 2 out of 10 items all parameter R^ values were below 1.01, with a total of 35 out of 60 parameter estimations yielding R^ values below 1.01. Traceplot inspections show a substantial autocorrelation for all items, except for Item 2. Across all model parameters, Item 2 has a minimum effective sample size (ESS) of 2561 (Parameter m_A). Other items have a considerably smaller minimal ESS in the range of 408 (m_C in Item 4) to 1470 (s_A in Item 1). All values of R^ (Table A1) and ESS (Table A2) can be found in the Appendix.

We examined the fit of the model by comparing model implied and empirical response accuracy, RT distributions, and conditional accuracy functions (CAFs), i.e. mean accuracies conditional on RT quantiles with posterior predictive checks.

Response accuracy prediction was very high with a correlation of r = 0.999 between empirical response accuracy and mean response accuracy in posterior predictive samples. Bias was < 0.01 for 9 out of 10 items, and 0.037 for Item 10. All empirical response accuracy rates are covered by the corresponding .80 credible interval of the predicted response accuracy rate.

The item distributions of RTs, conditional on response type, are shown in Figure 2 where the empirical densities are compared to the densities from posterior predictive samples. The RT distributions were captured very well for relatively fast responses from a larger number of participants, e.g., correct responses for the easier Items 1 and 2. Mismatches and larger variations in the predictions can be seen when responses were scarce (e.g., only two incorrect responses in Item 1) and when empirical data is bimodal. The bimodality originates in the time-limit of 30 seconds, that led participants to give a response shortly before the time ends, yielding mostly incorrect responses. Hence, bimodal distributions are more dominant for incorrect responses, most notable in Items 10, 6, and 4 where the model predictions deviate significantly from the empirical data. Predicted distributions for correct responses fit the data rather well.

Figure 3 shows the item-wise conditional accuracy functions (CAFs) calculated from the data and the posterior predictive samples. In the empirical data, three items show a negative relationship between RT and accuracy (Items 3, 4 and 9), that means slower responses tend to be less correct, five items show an inverted-U-shaped relationship (Items 2, 5, 6, 8 and 10), and the other two a mostly constant accuracy across RTs (Items 1 and 7). While the variation of model predictions can become quite large, the overall shape and magnitude of the conditional accuracy is well captured for all items. The 0.90 prediction interval does not include the data for only two points: one in Item 6 and one in Item 10.

The posterior medians as point estimates for each parameter and item are shown in Table 4. Regarding m_A, put on the scale of accumulated information by taking the exponential, the largest value corresponds to the easiest item that was solved by 99% of the participants and the smallest value corresponds to the hardest item (Item 6). Deviations of an ordering by solution frequency result from an interplay of effective speed α (represented by Parameters m_B and s_B) and stopping thresholds c₂ (represented by Parameters m_C and s_C). E.g., according to the model Item 4 is predicted to be easily solvable with expected maximum ability values around 18.91, well above the solution threshold c₁ = 10. However, a relatively large expected c₂ results in an earlier stop of the solution process and hence more incorrect responses. A correspondence of response times and parameters governing effective speed α (m_B and s_B) is also palpable as the item with the fastest responses (Item 2) yields the highest m_B and the item with the slowest responses (Item 10) yields the lowest values for m_B.

The time-limit of 30 seconds is challenging for the model. It occurred that participants responded just a few seconds before the time limit which resulted in bimodal empirical RT distributions. This suggests that participants checked for other, possibly better solutions before making their final response. The model does not account for this: it is predicted that the correct response is given only once the threshold is hit and that a incorrect response is given only once the progress is too slow. A potential integration of a time-limit in the model, e.g., by censoring the predicted RTs at the time-limit, could explain bimodality in the latency of incorrect responses but not for the correct responses that occur in the data, presumably from informed or random guessing. Despite these limitations, the model fits accuracy and RT data reasonably well and accurately reflects multiple forms of the relationship between RT and accuracy.

The model was also fit to matrix reasoning data from Myszkowski et al. (2022), who collected responses from N = 555 adult participants to Big Five personality items and a progressive matrices test in a low-stakes context. The 17 matrix items were automatically generated with the IMak R package (Blum & Holling, 2018). Data were collected in an online study and total screen times were recorded for the IMak items in full seconds. The IMak test was reported to be reliable, its validity was examined using ability estimates from an intelligence test that has been administered to all participants before the study. The dataset, as well as test information and the original analysis are available in the repository of the original study at Myszkowski and Storme (2022). Our analysis code, estimation results and diagnostic plots are available at Much and Ranger (2026).

Before fitting the data to the DBA model, we screened the data for outliers in the timing data. This was necessary because of the occurrence of unusually long screen times, presumably including longer non-response time in the online study when participants were distracted from working on the test. For each item, participant data were removed when the RT exceeded the range of 2.5 interquartile ranges from the upper or the lower quartile. This reduced the sample size to a number between 519 and 552 responses for each item. The RTs were rescaled with a factor of 1/6 to calibrate them with the RTs achievable by the model within the parameter range examined in the simulation study.

Again, the model parameters were estimated with estimator BH, with 40 histogram bins. Starting values were randomly drawn from uniform distributions around estimates from estimator ML1. Due to convergence issues and poor estimator efficiency with the setup used in the simulation study and the first empirical illustration, we extended the number of iterations to 40,000. Five thousand (5,000) iterations were discarded as burn-in.

Convergence was satisfactory for all parameters in all items. All R^ values were smaller than 1.10. For 8 out of 17 items, all parameter R^ values were below the stricter criterion of 1.01 (Vehtari et al., 2021), with a total of 78 out of 102 parameter estimations yielding R^ values below 1.01. Traceplot inspections reveal, however, substantial autocorrelations, most notably in Items 1, 13, and 17, for which minimum effective sample sizes were below 1000. Sufficiently low autocorrelations can be found for Items 5, 6, and 15, for which even the minimal effective sample sizes were larger than 1500. All values of R^ (Table A3) and ESS (Table A4) can be found in the Appendix.

Model fit was again examined with posterior predictive checks. Response accuracy prediction was very high with a virtually perfect correlation of r = 0.999 between empirical response accuracy and mean response accuracy in posterior predictive samples. Bias was < 0.006 for all items. All empirical response accuracy rates are covered by the corresponding .80 credible interval of the predicted response accuracy rate.

Item-wise conditional RTs distributions are shown in Figure 4. They are captured quite well by the model predictions. However, there is a larger offset for the correct responses on Item 12, which could only be solved by 121 participants (22.7%), and a larger offset for the correct responses on Item 16, that shows the largest difference in RT means between correct and incorrect responses (101.76s vs. 60.51s). Regarding the incorrect responses, some empirical distributions (Items 13 and 16) show a concentration of responses at very short RTs. This suggests a significant number of fast errors.

One type of fast errors are rapid guesses or non-responses that reflect strong disengagement. In the DBA, if the initial accumulation rate is already below the disengagement threshold c₂ , the DBA predicts zero-RTs. This is a reflection of maximum disengagement. In Figure 5, the posterior predicted rates of these events are plotted for each item. There is an overall increase across the test with nearly no occurrences predicted for the first items, up to rates of around 0.06 for Items 16 and 17. When comparing this to the threshold-based disengagement indicator Response Time Effort (RTE, Wise & Kong, 2005), we find correlations of 0.834 and 0.977 between the mean predicted zero-RT rates per item and RTE values based on plausible thresholds between 4 and 15 seconds.

Item-wise CAFs are shown in Figure 6. Again, Items 12 and 16 as well as Item 15 stand out as exceptions to a generally good fit of conditional accuracy. For most items, there is a positive correlation of RT and accuracy, with some inverted-U-shaped forms for Items 4, 7, and 13, that are well captured by the DBA model predictions. Misfitting items feature a step-like form indicating a sudden rise in the solution probability that cannot be accounted for by the model.

Posterior medians as point estimates for each parameter and item are shown in Table 5. The parameters governing the effective ability μ, m_A and s_A, show a small variance between items, with m_A estimates ranging from 2.31 to 2.55, translating to expectation values for μ of 10.04 and 12.76, and small estimates for the standard deviation within items, s_A, between 0.002 to 0.15. However, again the largest estimates for m_A correspond to the easiest Items (3, 4, and 5) while for the hardest Item (12), m_A is estimated with the lowest value. The correspondence between parameters governing effective speed α, m_B and s_B, is not as straightforward due to larger variations s_B and the effect of stopping thresholds c₂, reflected by model parameters m_C and s_C.

Myszkowski et al. (2022) analyzed the data with the dynamic speed joint hierarchical model of van der Linden and Fox (2016) and could achieve an acceptable model fit. An analysis of the empirical RT distributions, however, indicate a violation of the model’s log-normality assumption. In this situation, it seems reasonable to fit the data to a more flexible model in terms of RT distributions.

While the DBA is flexible to accommodate for shifts in RT distributions between correct and incorrect responses, it has its limitations with larger time differences. Predictions are drawn towards the RT distribution of the more frequent response. Regarding smaller offsets for incorrect responses, the variation of model predictions covers concentrations of fast responses very well, even though individual predictions fail to fully capture the resulting shape of the distribution. The DBA predicts zero-RTs when a test-taker disengages before the information accumulation starts. An extension of our baseline model that includes a transformation of these zero-RTs to realistic non-response times could even improve the prediction of outstanding fast errors. However, even without this extension, the rate of predicted zero-RTs can be interpreted as a model-based indicator of disengagement. This indicator correlates highly to the threshold-based RTE and was capable to detect the common finding of decreasing test-taking engagement over the course of a test.

Regarding the estimated parameter values, the results indicate a surprisingly small variance for the parameters that relate to a person’s maximum ability. This concerns both the between-trial variance s_A and the variance of m_A between the items. It is noticeable that the estimates for m_A relate to effective parameter values of μ that are close to the fixed solution threshold c₁ = 10. This suggests that the model more efficiently represents the variance in the data with variances of processing speed and persistence. Assuming the model is valid, this indicates that given enough time and persistence, virtually everyone would be able to solve the items.