Grand Challenge 3 Evaluation Results

Pose predictions were evaluated in terms of symmetry-corrected root-mean-square deviations (RMSD, Å). Stage 1A is a cross-docking challenge while Stage 1B is a self- docking challenge. In the graphs, the pull-down menu allows display of the results for all submissions for each compound (e.g., CatS_1), the average over all compounds (Mean over all), or the median over all compounds (Median over all). The three buttons above the menu in the graphs allow display of the RMSD values for lowest RMSD pose among the maximum of five in each submission (Closest); of the average RMSD value across all five (Average); or of the RMSD value for the top-scoring pose in each submission (Pose 1). The tables and csv files provide the statistics averaged over ligands.

This section presents metrics of the ability of the predictions to correctly rank ligands by affinity. The rankings were evaluated in terms of the Kendall's τ, Spearman's ρ, and estimated binding energies for the free energy sets were additionally evaluated in terms of centered root-mean-square error (RMSEc, kcal/mol) and Pearson's r. Uncertainties in these statistics (e.g., Kendall's τ Errors in the table) were obtained by recomputing them in 10,000 rounds of resampling with replacement, where, in each sample, the experimental IC50 or Kd data were randomly modified based on the experimental uncertainties. Experimental uncertainties are added to the free energy, ΔG, as a random offset δG drawn from a Gaussian distribution of mean zero and standard deviation RTln(Ierr). In this evaluation, the value of Ierr was set to 2.5.

For the kinases, a number of experimental Kd values were reported as ≥10 µM, making them difficult to include in standard metrics of ranking accuracy, so these cases were excluded from these affinity ranking assessments. However, they are included in the Active/Inactive Classification assessments, below.

A number of experimental Kd values were reported as ≥ 10 µM, making them difficult to include in standard metrics of ranking accuracy. Instead, for the full set of compounds, we used the Matthews Correlation Coefficient, a classification statistic, to evaluate the ability of prediction methods to differentiate between the highest and lowest affinity ligands. In effect, we considered the N_a compounds with measured Kd values <10 µM as active, and the N_i ones with measured Kd values >10 µM as inactive. We similarly assigned the N_i lowest-ranked ligands in each submission as inactive, and the top- ranked N_a as active, and used the Matthews statistic to assess how well the predictions did at classifying the ligands.

In order to test methods in a setting where errors in the ligand poses do not contribute to errors in the affinity predictions, all submissions from Stage 1 and 2 were reevaluated for the CathepsinS dataset using only the 19 ligands for which crystallographic poses had been provided in Stage 2 (CatS 1 to CatS 24, excluding CatS 7, 9, 14, and 21). The rankings of compounds by affinity were evaluated in terms of the Kendall's τ, Spearman's ρ, and the free energy sets were additionally evaluated in terms of centered root-mean-square error (RMSE_c, kcal/mol) and Pearson's r. Uncertainties in these statistics (e.g., Kendall's t Errors in the table) were obtained by recomputing them in 10,000 rounds of resampling with replacement, where, in each sample, the experimental IC50 data were randomly modified based on the experimental uncertainties. Experimental uncertainties are added to the free energy, ΔG, as a random offset δG drawn from a Gaussian distribution of mean zero and standard deviation RTln(I_err). In this evaluation, the value of I_err was set to 2.5.