Building evidence on AI-assisted review in public funding
The Neotec shadow experiment with CDTI
Public funding agencies are exploring whether AI can improve how they assess applications. But understanding its value requires more than testing a model’s accuracy: agencies also need to know how assessors respond to its advice and whether it helps address the problems they face.
IGL supported Spain’s innovation agency, CDTI, in exploring these questions within Neotec, its funding programme for early-stage technology-based companies. The starting point was a practical challenge: how to improve the assessment of proposals’ potential social impact.
Together with an academic team, we developed a “shadow experiment” involving 116 participants, 348 real proposals and 828 evaluations. Alternative assessment approaches were tested after the official evaluation had concluded, without affecting applicants’ scores or funding decisions.
Predicting social impact proved difficult across all evaluator groups. External social-impact experts did not substantially outperform other assessors, pointing to broader challenges in defining, measuring and anticipating the societal effects of early-stage innovation.
AI-generated information produced small but consistent improvements in evaluators’ predictive performance. Its value was as a complementary signal that could help people reconsider their judgement, although the predictive findings should be interpreted cautiously given the limited validation sample.
Evaluators used AI selectively. They were more likely to revise their assessments when its input differed substantially from their initial judgement and, in historical cases, was closer to observed outcomes. Experienced Neotec assessors were particularly unlikely to revise their scores.
Presentation also mattered. Explicit explanations increased the proportion of scores revised from 14% to 24%, mainly among external experts. Evaluators were more receptive to suggestions to lower scores than to raise them, highlighting how AI support could unintentionally make assessments more conservative.
The project depended on collaboration between CDTI, academic researchers and IGL as a knowledge broker. Adapting the design, addressing confidentiality concerns and securing staff participation required sustained coordination, trust and institutional support. The “shadow experiment” approach allowed us to learn under realistic conditions while keeping the experiment separate from public funding decisions.
“The collaboration with IGL was critical in developing strategies to navigate the practical and institutional challenges involved in running an experiment of this kind, while also supporting the connection between the academic and operational sides of the project.”
– CDTI, Spanish Innovation Agency
What can you find in this policy brief?
This brief explains the Neotec experiment’s design, findings and practical lessons. It explores the potential and limits of AI-assisted assessment, how human behaviour shapes its usefulness, and the organisational conditions that enable public agencies to experiment and learn before introducing changes.
-
ProjectNeotec Shadow Experiment