The SAMPL experiment
SAMPL is an attempt at prospectively testing protein and ligand modeling. Ideally this would consist of experiments conceived to distinguish between competing ideas or methods and perhaps over time we shall get there. For now, this is an analysis of methods applied to data not seen by participants, a 'blind' assessment. We make no claims against the limitations of such an attempt, believing it more important to light a candle than curse the darkness. And blind tests do help us avoid the tendency to bias our theories and approaches to known answers. They also provide a more realistic "real world" setting for methods.
With SAMPL we intend to avoid "who won, who lost". On such a small sample size, no such statistically sound pronouncement could be made anyway. Rather, we see this as an opportunity for groups to test their methods, learn from the experience and share lessons learned.
Previous SAMPLs
At CUP8, in 2007, we had a small taste of blinded prospective evaluation data sets on vacuum-water transfer and protein-ligand binding. We were fortunate to have Vijay Pande, Peter Guthrie and Anthony Nicholls discuss transfer energies and Hideaki Fujitani, Scott Brown and Marti Head discuss protein-ligand binding predictions. That was enough to demonstrate the value of this exercise. In addition, the work on solvation prediction appeared in J. Med. Chem. in 2008 (Nicholls et. al, JMC 2008 51(4):769-779 http://pubs.acs.org/doi/abs/10.1021/jm070549%2B).
SAMPL-1, held in winter 2007-2008 consisted of two sets of protein-ligand binding data, generously provided by Abbott Labs and Vertex Pharmaceuticals, and sixty three vacuum-water transfer energies, courtesy of Peter Guthrie at the University of Ontario, and CCG. The two sets of protein-ligand data each have between twenty and sixty active compounds, each with a protein-ligand crystal structure. The protein-ligand assessment consisted of three parts: virtual screening, pose prediction and affinity prediction. The vacuum-water transfer energies are provided by Peter Guthrie. SAMPL-1 had over 50 participants who submitted over 200 data samples. Approximately half the participants were from industry and half were from academic or government labs. The results of SAMPL-1 were discussed March 19th at CUP9, including presentations from Vijay Pande of Stanford, Marti Head of GSK, Enrico Purisima of BRI-NRCC and Dave Mobley of UCSF. A reprise of the results was also presented at the GRC, ACS meetings in the US and the ICCS meeting in Europe. Finally, Peter Guthrie has organized a recently-published special issue of JPC that focuses on the vacuum-water transfer results from SAMPL-1. (http://pubs.acs.org/toc/jpcbfk/113/14)
SAMPL-09
SAMPL-09 consists of transfer energy and tautomer ratio predictions. Once again, Peter Guthrie has organized the transfer energy data, and Peter Taylor, formerly from Astra-Zeneca, has gathered the tautomer ratio data.
As with previous years, we expect SAMPL-09 will result in one or more publications. We will follow an 'opt-in' policy for any publication, i.e. the results from any participants can be used but attribution is by permission. We hope this may encourage some who, perhaps correctly, feel they have more to lose than gain. Simply put, participants can choose to remain anonymous. Details can be found in the sign-up agreement.
More details, including expected formats for entries, cut-off dates for participation and expected degrees of confidentiality are available by signing up. The deadline for submission of entries is May 18, 2009. Discussion of SAMPL-09 results will take place on June 18th and 19th at a workshop at McGill University in Montreal. More details can be found on the SAMPL09 Workshop Info page, and registration is now open.
We hope SAMPL continues to be useful and regular event. There are many possible variants we plan to try in future assessments, such as providing partial information, e.g. a subset of actives or binding modes, more typically of an industrial project setting. Additional physical properties, such as tautomer ratios, pKas, thermodynamic data could be sought. We strongly believe a blinded assessment component to our field will help us judge progress and hope you both agree and can contribute.
Kind Regards,
Geoff Skillman & Anthony Nicholls
OpenEye Scientific Software, Inc.
