Assay-Based Machine Learning: Rethinking Evaluation in Drug Discovery

Abstract

The success of machine learning (ML) in drug discovery relies on modeling and evaluation strategies that accurately reflect the conditions under which compounds are tested in experimental practice. However, we found that experimental bioactivity data in public repositories like ChEMBL is highly clustered by assay origin, in both molecular structures and reported activity values. As a result, common splitting strategies, such as random or scaffold splits, still lead to more than 89% overlap in assay origin between training and test sets across multiple ChEMBL-derived subsets. This latent overlap enables simple baselines, such as predicting the mean activity of an assay from the training data, to perform unexpectedly well on test compounds originating from the same assay. To address this issue, we propose an ML framework that accounts for the inherent structure of, especially freely available, aggregated experimental data in drug design. We implement this approach through: (1) data splitting that allocates entire assays to either training or test sets, (2) evaluation metrics that assess ranking performance within individual assays rather than absolute prediction accuracy across heterogeneous experiments, and (3) set-based ranking models trained specifically on compound sets drawn from the same assay rather than random sets. Evaluating our approach on three datasets derived from ChEMBL, we demonstrate that ranking models trained on intra-assay sets consistently rank compounds of unseen assays more reliably than ranking models trained on arbitrary sets and direct affinity prediction models. Furthermore, we demonstrate that the correlation over the entire test set is much higher than the average correlation within experimental sets. Our findings indicate that ML methods for drug discovery should prioritize intra-assay ranking capability over absolute value prediction when working with aggregated experimental data.

@article{grossAssayBasedMachineLearningRethinking2026,
 author = {Groß, Joschka and Backenköhler, Michael and Volkamer, Andrea},
 doi = {10.1039/D6DD00472E},
 eprint = {https://pubs.rsc.org/dd/article-pdf/doi/10.1039/D6DD00472E/15052126/d6dd00472e.pdf},
 issn = {2635-098X},
 journal = {Digital Discovery},
 month = {11},
 title = {Assay-Based Machine Learning: Rethinking Evaluation in Drug Discovery},
 url = {https://doi.org/10.1039/D6DD00472E},
 year = {2026}
}