.svg){kind=small}
AI-ready data starts with data quality
Quantifying the impact of human and AI agent data curation methods in biomedical research
Executive Summary
Artificial Intelligence is rapidly becoming a core tool for biomedical research. Scientists increasingly rely on Large Language Models (LLMs) to search, summarize, classify, and analyze biological datasets. However, the quality of AI-generated answers is fundamentally constrained by the quality of the underlying data.
To evaluate the relationship between data curation quality and the accuracy of AI-driven cohort identification, Rancho BioSciences conducted a controlled benchmarking study comparing three representations of the same underlying single-cell datasets.
Using identical scientific questions and identical AI-driven retrieval workflows, we measured the ability of LLM agents to correctly identify relevant samples and studies.
Results demonstrated a clear relationship between data curation quality and the accuracy of AI driven retrieval and cohort identification:
| Single Cell Data Alliance |
scBaseCount | Raw GEO | |
|---|---|---|---|
| macro-Recall | 0.94 | 0.80 | 0.58 |
| macro-Precision | 0.95 | 0.73 | 0.82 |
| macro-F1 Score | 0.92 | 0.70 | 0.60 |
| macro-Accuracy | 0.99 | 0.93 | 0.88 |
The results reveal a clear progression in metadata quality and retrieval accuracy: raw metadata performed the worst, agentic curation improved performance, and SME-curated metadata performed best. Single Cell Data Alliance metadata improved overall macro-F1 score by more than 50% compared with raw GEO metadata and substantially outperformed agentic curation approaches.
These findings provide quantitative evidence that investments in expert curation translate directly into cleaner, more accurate data, and therefore better AI outcomes.
For organizations building AI-driven research capabilities, these results suggest that the most effective strategy is not to replace experts with AI, but to use AI to amplify expert capabilities. While AI can dramatically improve the speed and scale of data curation, expert oversight remains essential for ensuring biological accuracy, contextual understanding, and data quality.