Revista de Economia e Sociologia Rural
https://revistasober.org/article/doi/10.1590/1806-9479.2025.290629
Revista de Economia e Sociologia Rural
ORIGINAL ARTICLE

Enhancing the use of impact evaluation results: a multi-case study in agricultural research organizations

Potencializando o uso dos resultados de avaliações de impacto: estudo multicaso em organizações de pesquisa agrícola

Daniela Maciel Pinto; Adriana Bin

Downloads: 0
Views: 17

Abstract

Impact evaluation has historically been the primary method for assessing agricultural research. However, the practical use of its results remains underexplored in the literature. Addressing this gap, this study investigates the use of impact evaluation results in agricultural RD&I organizations through a multi-case analysis of eight organizations across three continents. Grounded in the social responsibility and ethics of research and evaluation, this study addresses the question: “How are impact evaluations in agricultural RD&I organizations conducted, and how are their results utilized to enhance strategic decision-making and innovation?” To answer this question, the AGRIUM model was developed and applied, focusing on objectives, methods, impact dimensions, uses, and stakeholders. The findings indicate that neither experience with evaluations nor the diversity of evaluated dimensions directly influence the utilization of results. Instead, three categories of factors emerged: (1) structural and organizational, such as integration into strategic planning; (2) operational, including evaluation quality, communication, and timeliness; and (3) team literacy and stakeholder pressures. Establishing a well-defined process, including information management, documentation, feedback practices, and monitoring recommendations, was identified as critical not only for driving impactful agricultural research but also for informing policy decisions and strengthening institutional strategies.

Keywords

research and development, management of technological innovation and R&D, evaluation result utilization, impact evaluation

Resumo

Resumo: A avaliação de impacto tem sido historicamente o principal método para avaliar a pesquisa agrícola. Contudo, o uso prático de seus resultados permanece pouco explorado na literatura. Este estudo investiga este uso em organizações de PD&I agrícola, por meio de uma análise multicaso de oito instituições em três continentes. Embasado na responsabilidade social e ética da pesquisa e avaliação, responde à pergunta: 'Como as avaliações de impacto em organizações de P&D agrícola são realizadas e como seus resultados são utilizados para aprimorar a tomada de decisão estratégica e a inovação?' Para isso, desenvolveu-se e aplicou-se o modelo AGRIUM, focado nos objetivos, métodos, dimensões de impacto, usos e stakeholders. Os resultados indicam que nem a experiência com avaliações nem a diversidade de dimensões avaliadas influenciam o uso dos resultados. Três categorias de fatores de influência emergem: (1) estruturais e organizacionais, como a integração ao planejamento estratégico; (2) operacionais, incluindo qualidade e comunicação das avaliações; e (3) literacia da equipe e pressões de stakeholders. Estabelecer um processo bem definido, com gerenciamento de informações, documentação, práticas de feedback e monitoramento, foi identificado como essencial não apenas para potencializar pesquisas agrícolas impactantes, mas também para subsidiar decisões estratégicas e políticas institucionais.

Palavras-chave

pesquisa e desenvolvimento, gestão da inovação tecnológica e P&D, utilização de resultados de avaliação, avaliação de impacto

Referências

AgResearch. (2021). Statement of Corporate Intent: 2023-2028. Retrieved in 2024, September 25, from https://www.landcareresearch.co.nz/publications/statement-of-corporate-intent/

Alkin, M. C., & King, J. A. (2016). The historical development of evaluation use. The American Journal of Evaluation, 37(4), 568-579. http://doi.org/10.1177/1098214016665164

Bardin, L. (2011). Content analysis. São Paulo: Edições 70.

Barret, D., Blundo Canto, G., Dabat, M.-H., Devaux-Spatarakis, A., Faure, G., Hainzelin, E., Mathé, S., Temple, L., Toillier, A., & Triomphe, B. (2017). Guide méthodologique ImpresS. Évaluation ex post des impacts de la recherche agronomique dans les pays du Sud. Retrieved in 2024, September 25, from https://agritrop.cirad.fr/586223/

Blundo-Canto, G., Triomphe, B., Faure, G., Barret, D., De Romemont, A., & Hainzelin, E. (2019). Building a culture of impact in an international agricultural research organization: process and reflective learning. Research Evaluation, 28(2), 2.

CGIAR. (2025) CGIAR Results Dashboard. Retrieved in 2024, September 25, from https://www.cgiar.org/dashboards/

CGIAR. (2020). SPIA Approach to Impact Assessment for CGIAR. Retrieved in 2024, September 25, from https://iaes.cgiar.org/sites/default/files/pdf/SPIA%20Technical%20Note%208.pdf

Colinet, L. (2021). ASIRPA: Societal impact of research. Retrieved in 2024, September 25, from https://www6.inrae.fr/asirpa_eng/ASIRPA-project/Method

Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. New Directions for Evaluation, 2014(141), 7-23. http://dx.doi.org/10.1002/ev.20076

Creswell, J. W. (2013). Qualitative inquiry and research design: choosing among five approaches (3rd ed.). London: SAGE Publications.

Creswell, J. W. C., & Creswell, J. D. (2018). Research design: qualitative, quantitative, and mixed methods approaches (5. ed). Retrieved in 2024, September 25, from http://gen.lib.rus.ec/book/index.php?md5=7f211eac2e39650c5e4506f6a43df875

Cruz, G. M., & Miranda, S. H. G. (2022). Reflexões para uma pecuária mais sustentável. Revista de Política Agrícola, 31(4), 152.

CSIRO (2020). Impact Evaluation Guide. Retrieved in 2024, September 25, from https://www.csiro.au/en/about/corporate-governance/ensuring-our-impact/auditing-our-impact/2022-impact-assessment

Deniston, O. L. (1980). Whether evaluation–whether utilization. Evaluation and Program Planning, 3, 91-94.

Díez, M., Izquierdo, B., & Malagón, E. (2016). Increasing the Use of Evaluation Through Participation: The experience of a rural sustainable development plan evaluation. Environmental Policy and Governance, 26(5), 366-376. http://dx.doi.org/10.1002/eet.1711

Douthwaite, B., Proietti, C., Polar, V., & Thiele, G. (2023). Outcome Trajectory Evaluation (OTE): An Approach to Tackle Research-for-Development’s Long-Causal-Chain Problem. The American Journal of Evaluation, 44(3), 335-352. http://dx.doi.org/10.1177/10982140221122771

Eisenhardt, K. M. (2021). What is the Eisenhardt Method, really? Strategic Organization, 19(1), 147-160. http://dx.doi.org/10.1177/1476127020982866

Embrapa (2024a). Relatório de administração 2023. Brasília, DF: EMBRAPA.

European Commission (2023). Better regulation: guidelines and toolbox. Retrieved in 2024, September 25, from https://commission.europa.eu/law/law-making-process/planning-and-proposing-law/better-regulation/better-regulation-guidelines-and-toolbox_en

Evenson, R. E. (1982). Observations on Brazilian Agricultural Research and Productivity. Revista de Economia e Sociologia Rural, 20(3), 367-401.

Felt, U. (2018). Responsible Research and Innovation. In Sarah Gibbon, Barbara Prainsack, Stephen Hilgartner and Janette Lamoreaux (eds.) Handbook of Genomics, Health and Society. London/New York: Routledge.

Ferré, M., Blundo-Canto, G., Rodrigues, G. S., Ramírez-Gómez, M. M., Vedovoto, G. L., Agudelo-Chocontá, B.-E., Marques, D. V., Flores, R. M. V., Rodríguez-Borray, G.-A., de Souza, M. O., Goulet, F., Vásquez-Urriago, A., Sánchez-Lozano, J.-I., Pinto, D. M., Zambrano-Moreno, G.-S., Londoño-Arias, M.-A., Zapata-Tamayo, C.-A., & Romémont, A. (2025). Culture of impact in agricultural research organisations: What for and how? Research Policy, 54(1), 105140. http://doi.org/10.1016/j.respol.2024.105140

FTeval. (2003). Evaluation standards in research and technology policy. Vienna, Austria: Plattform-Forschungs-und Rechnologieevaluieung GesbR.

Gertler, P. J., Martinez, S., Premand, P., & Rawlings, L. B. (2018). Avaliação de Impacto na Prática (2. ed). Washington, DC: World Bank Publications.

Hall, A., Sulaiman, V. R., Clark, N., & Yoganand, B. (2003). From measuring impact to learning institutional lessons: An innovation systems perspective on improving the management of international agricultural research. Agricultural Systems, 78(2), 213-241. http://doi.org/10.1016/S0308-521X(03)00127-6

Horton, D., & Mackay, R. (2003). Using evaluation to enhance institutional learning and change: Recent experiences with agricultural research and development. Agricultural Systems, 78(2), 127-142.

Mayne, J. (2015). Useful theory of change models. The Canadian Journal of Program Evaluation, 30(2), 119-142. http://doi.org/10.3138/cjpe.230

Mertens, D. M. (2016). Assumptions at the philosophical and programmatic levels in evaluation. Evaluation and Program Planning, 59, 102-108. http://doi.org/10.1016/j.evalprogplan.2016.05.010

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice. New York: Guilford Publications.

Joly, P.-B., Colinet, L., Gaunand, A., Lemarié, S., & Matt, M. (2016). Agricultural research impact assessment: Issues, methods and challenges. Retrieved in 2024, September 25, from https://hal.archives-ouvertes.fr/hal-01431457.

Julnes, G., & Mark, M. M. (1998). Evaluation as sensemaking: Knowledge construction in a realist world. New Directions for Evaluation, 1998(78), 33-52.

Lee, S., Díaz-Puente, J., & Vidueira, P. (2020). Enhancing rural innovation and sustainability through impact assessment: a review of methods and tools. Sustainability (Basel), 16(16), 6559. http://dx.doi.org/10.3390/su12166559

Mackay, R., & Horton, D. (2003). Expanding the use of impact assessment and evaluation in agricultural research and development. Agricultural Systems, 78, 143-165. http://dx.doi.org/10.1016/S0308-521X(03)00124-0

Midmore, P. (2017). The Science of Impact and the Impact of Agricultural Science. Journal of Agricultural Economics, 68(3), 611-631. http://dx.doi.org/10.1111/1477-9552.12242

Milzow, K., Reinhardt, A., Söderberg, S., & Zinöcker, K. (2019). Understanding the use and usability of research evaluation studies. Research Evaluation, 28(1), 94-107. http://doi.org/10.1093/reseval/rvy040

Morgan Jones, M., Grant, J., & Rand, E. (2013). Making the grade: methodologies for assessing and evidencing research impact. In D. Cope (Ed.), 7 Essays on Impact. DESCRIBE Project Report for Jisc. Exeter: University of Exeter.

Morgan Jones, M., Manville, C., & Chataway, J. (2022). Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future. The Journal of Technology Transfer, 47(3), 722-746.

Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2015). Handbook of Practical Program Evaluation. San Francisco: Wiley.

OECD. (2023). Developing evaluation capacities. Retrieved in 2024, September 25, from https://www.oecd.org/en/topics/development-co-operation-evaluation-and-effectiveness.html.

Pareja, M., Bervejillo, J., Bianco, M., Ruíz, A., & Torres, A. (2011). Evaluación de los impactos económicos, sociales, ambientales e institucionales de 20 años de inversión en investigación e innovación agropecuaria por parte del Instituto Nacional de Investigación Agropecuaria (INIA)–Uruguay: Resumen ejecutivo. Montevideo: INIA.

Parker, C., Scott, S., & Geddes, A. (2019). Snowball Sampling. Retrieved in 2024, September 25, from https://eprints.glos.ac.uk/id/eprint/6781

Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage publications.

Patton, M. Q. (2011). Essentials of utilization-focused evaluation. California: Sage publications.

Patton, M. Q., & Horton, D. (2009). Utilization-focused evaluation for agricultural innovation. Netherlands: ILAC Brief.

Pena Junior, M. A. G., & Francozo, M. A. S. (2023). O futuro da agricultura brasileira: 10 visões. Retrieved in 2024, September 25, from https://www.alice.cnptia.embrapa.br/alice/handle/doc/1153216

Pereira, C. N., & Castro, C. N. (2020). O Sistema Nacional de Pesquisa Agropecuária e a análise dos investimentos no Fundo Setorial do Agronegócio. Revista de Economia e Sociologia Rural, 58, e181041.

Pinto, D. M., & Bin, A. (2024) Use of evaluation results in S&T and R&D: what does literature tell us? Rio de Janeiro: MetAvaliação.

Pinto, D. M., Bin, A., & Rodrigues, G. S. (2025). Trends and challenges in the impact evaluation of agricultural research: a systematic literature review. Cadernos de Ciência & Tecnologia. [No prelo]

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. The American Journal of Evaluation, 29(4), 443-459.

Rodrigues, G. S., Campanhola, C., & Kitamura, P. C. (2003). An environmental impact assessment system for agricultural R&D. Environmental Impact Assessment Review, 23(2), 219-244. http://doi.org/10.1016/S0195-9255(02)00097-5

Rogers, P. (2014). Theory of change: Methodological briefs-impact evaluation. Florence: UNICEF. Retrieved in 2024, September 25, from https://ideas.repec.org/p/ucf/metbri/innpub747.html

Saari, E., & Kallio, K. (2011). Developmental Impact Evaluation for Facilitating Learning in Innovation Networks. The American Journal of Evaluation, 32(2), 227-245. http://dx.doi.org/10.1177/1098214010387658

Spaapen, J. B. (2015). A new evaluation culture is inevitable. Organic Farming, 1(1), 36-37. http://doi.org/10.12924/of2015.01010036.

Stockmann, R., Meyer, W., & Szentmarjay, L. (2022). The institutionalisation of evaluation in the Americas. Cham: Springer Nature.

Stockmann, R., Meyer, W., & Taube, L. (2020). The institutionalisation of evaluation in Europe. Cham: Springer Nature.

Stratton, S. J. (2021). Population research: convenience sampling strategies. prehospital and disaster medicine. Prehospital and Disaster Medicine, 36(4), 373-374. http://doi.org/10.1017/S1049023X21000649

Turner, J. A., Guesmi, B., Gil, J. M., Heanue, K., Sierra, M., Percy, H., Bortagaray, I., Chams, N., & Milne, C. (2022). Evaluation capacity building in response to the agricultural research impact agenda: Emerging insights from Ireland, Catalonia (Spain), New Zealand, and Uruguay. Evaluation and Program Planning, 94, 102127.

Van der Most, F. (2010). Use and non-use of research evaluation: a literature review. Retrieved in 2024, September 25, from http://wp.circle.lu.se/upload/CIRCLE/workingpapers/201016_vanderMost.pdf

Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 5(5), 426. http://dx.doi.org/10.2307/3109916

Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? The American Journal of Evaluation, 19(1), 21-33

Weißhuhn, P., Helming, K., & Ferretti, J. (2018). Research impact assessment in agriculture: a review of approaches and impact areas. Research Evaluation, 27(1), 36-42.

White, T., Percy, H., & Small, B. (2018). Creating an evaluation culture through capacity building: A new frontier in a science organisation. Evaluation Matters–He Take Tō Te Aromatawai, 4, 111-133. http://doi.org/10.18296/em.0031

Yin, R. K. (2009). Case study research: design and methods (4. ed). California: Sage.

AGROSAVIA. (2024). Balance Social. Retrieved in 2024, September 25, from https://www.agrosavia.co/sociedad/balance-social.

Embrapa. (2024b). Social Report. Retrieved in 2024, September 25, from https://www.embrapa.br/en/balanco-social.
 


Submetido em:
25/09/2024

Aceito em:
25/03/2025

6834c367a953956a766c7a05 resr Articles
Links & Downloads

resr

Share this page
Page Sections