Quality of Science: Evaluative Evidence from CGIAR Portfolio Summary

CGIAR is a global research partnership for a food-secure future. The 2020 CGIAR reform restructured CGIAR’s partnerships, knowledge, and operations to create One CGIAR. Its mission is to end hunger by 2030 through science to transform food, land, and water systems in a climate crisis aimed at contributing to the achievement of the Sustainable Development Goals (SDGs).

IN 2023, the United Nations created an independent Scientific Advisory Board to advise its leaders on breakthroughs in science and technology and how to harness the benefits of these advances and mitigate potential risks to support efforts to achieve the SDGs. The successful evaluation of quality of science (QoS) is an integral part of this process and the importance of QoS as a formative evaluation criterion was explicitly emphasized for the first time in CGIAR evaluations and reviews in 2020.

The main purpose of this synthesis document is to showcase evidence collected from evaluating QoS in CGIAR evaluations and reviews, irrespective of whether a designated QoS evaluation criterion was used explicitly or implicitly. It contains the following sections summarized by QoS dimensions (design, inputs, processes, and outputs) and relevant recommendations: evaluative reviews of twelve 2020 CRPs; a review of the 2021 synthesis report; reviews of the GENDER, Genebank and Big Data in Agriculture Platform evaluations; and a summary of three case studies used to apply the QoS dimensions at the 2023 workshop. Both quantitative and qualitative indicators from the new evaluation guidelines were used. The main objective was to illustrate the use and value of the QoS criterion in practice. It was explicitly used in the 2020 CRP reviews (without the design dimension), but implicitly used in the 2022‒23 platform reviews.

Despite different evaluation frameworks being used for CGIAR evaluations and reviews during the past decade, the synthesis generated a wealth of useful information on QoS which is crucial for improving the contribution of CGIAR research to the achievement of the SDGs. Synthesis of the design dimension highlighted the importance of within and between program and platform linkages and communities of practice to enhance the ability to take advantage of complementarities and synergies as well as the need to improve the utility of theories of change as planning tools. Synthesis of the inputs dimension emphasized the value of adequate skills, diversity (including gender), and strong partnerships for implementation of planned research as well as the need for adequate and certain funding to support delivery of outputs and outcomes. Synthesis of processes highlighted the importance of legitimacy and trust in stakeholder partnerships and networking and well-defined roles and responsibilities at both research and management levels for progress and success. Finally, the synthesis of outputs highlighted the generation of an impressive quantity and high quality of outputs: scientific publications, technical documents, physical outputs such as improved crop varieties, methodologies, tools, models, and communication products that produced recognizable and measurable international public goods. Some emphasis was placed on analyzing scientific publications which revealed the untapped potential of bibliometrics as a component of quantitative and qualitative evaluation of QoS in an R4D context.

This retrospective analysis and synthesis of the QoS dimensions of CGIAR evaluation documents generated from 2020 to 2023 has demonstrated a crucial role as well as the utility of QoS as an important evaluative criterion in R4D programs and projects. By showcasing the growing body of evidence highlighting the value of applying the QoS criterion to the evaluation of R4D programs and projects, it is more likely that evaluators and subject-matter experts will adopt the criterion in future evaluations of such programs and projects.