Häckl, S, Koch, A, Lasch, F. Empirical evaluation of the implementation of the EMA guideline on missing data in confirmatory clinical trials: Specification of mixed models for longitudinal data in study protocols. Pharmaceutical Statistics. 2019; 18: 636– 644. doi: 10.1002/pst.1964

In longitudinal studies data may be repeatedly measured and collected on the same subjects. When analysing these data the statistician has to consider the correlation between single measurements of the same subject. This can be done by using a mixed model for longitudinal data as a statistical model, which allows modelling the correlated measurements via a covariance matrix. Thereby, not only the correlation is considered, but also missing outcome data are implicitly imputed. To define a mixed model for longitudinal data a number of model characteristics need to be specified: (a) the fixed and repeated/random effects (b) the covariance matrix, (c) the testing method, (d) the computation method, (e) the estimation method or (f) the fallback strategy defining the handling of computation or convergence problems.

In confirmatory clinical trials, the pre-specification of the primary analysis model is a universally accepted scientific principle to allow strict control of the type-I-error. Otherwise, multiplicity problems and subsequently, the inflation of the type-I-error may occur. Consequently, both the ICH E9 guideline and the EMA guideline on missing data in confirmatory clinical trials require the explicit specification of the precise model settings to such an extent that the statistical analysis can be replicated. This requirement also applies to the above mentioned specifications when mixed models for longitudinal data – handling missing data implicitly – are used as the primary analysis.

To evaluate the compliance with the EMA guideline, the model specifications in clinical study protocols from development phase II and III submitted were elaborated between 2015 and 2018 to the Ethics Committee at Hannover Medical School under the German Medicinal Products Act, which planned to use a mixed model for longitudinal data in the confirmatory testing strategy.

Overall, 39 trials from different types of sponsors and a wide range of therapeutic areas were evaluated. While nearly all protocols specify the fixed and random effects of the analysis model (95%), only 77% give the structure of the covariance matrix used for modelling the repeated measurements. Moreover, the testing method (36%), the estimation method (28%), the computation method (3%) and the fallback strategy (18%) are given by less than half the study protocols. Not a single protocol specified all evaluation items as required by the EMA guideline. The fixed/random effects, the covariance structure and testing method were clustered as the most sensitive items. While only 31% of the study protocols specified all of these main items less than half of the analysed protocols (46%) neither specified all main items nor planned to do so by an additional statistical analysis plan (SAP).

Subgroup analysis indicates that these findings are universal and not specific to clinical trial phases or size of company. However, surprisingly a slightly higher proportion of protocols specifying the main items in phase II compared to phase III clinical trials (33% vs. 29%) was noted. Comparing the types of sponsor showed a significant difference between major pharmaceutical companies and minor sponsors in specifying the three main items (53% vs. 14%; p<0.05). This discrepancy is slightly less distinctive in phase III (46% vs. 15%; p=0.18) than in phase II (67% vs. 11%; p=0.09), although not reaching statistical significance.

In summary, while fixed and random effects are usually specified, a lack of statistical rigor in unambiguously specifying other influential parameters was observed, such as the covariance matrix or testing method. Additionally, some sponsors refer to the supplementing SAP for exact model specification. However, this constitutes a problem for ethics committees in assessing the primary analysis model entirely when the referenced SAP is not submitted along with the study protocol.

Altogether, our results show that guideline compliance is to various degrees poor and consequently, strict type-I-error rate control at the intended level is not guaranteed. Therefore, not only the exact specification of model parameters in the study protocol was suggested but also complementary provision of the statistical software, its version and the software code used to build the mixed model was highly recommended.