AI and its use in evaluation
Throughout the development of evaluation, there has been a strong tradition of using digital technology to enable better processes and products. Using technology in evaluation has been especially true within the past two decades with the advancements in computer-assisted data collection, analysis, and reporting.
At the frontier of tech-enabled evaluation has been a small cadre of practitioners using emerging technologies such as big data, predictive analytics, machine learning, and traditional artificial intelligence.
Many of these cutting edge technological applications for evaluation have been either limited test cases or high barriers of utilisation for the average evaluator. However, at the end of 2022, a significant inflection point for evaluation—and all knowledge-work sectors—occurred with the widespread exposure and use of GenAI technology, especially large language models (LLMs).
As a result, many professions and organizations have been attempting to respond to the disruption GenAI has caused—including in the evaluation sector. Much of this disruption is due to the significant advancement in AI model performance and significantly lower barriers to use, with little to no technical experience needed by users to obtain automated responses.
GenAI-powered evaluation function at UNFPA
UNFPA’s GenAI-powered evaluation strategy supports the existing priorities and values of the UNFPA evaluation function, notably strengthening accountability, evidence-based decision-making, and learning. It is responsive to changing global dynamics and seeks to help the evaluation function be more agile and adaptive amidst changing contexts. Finally, it intends to strengthen an organizational culture of evidence-based decision making for improved programming and organizational effectiveness.
Strategy goals
Internally, strengthen the UNFPA evaluation function by optimizing evaluation processes and products with ethical and responsible use of GenAI tools. Externally, co-lead the shaping of responsible and ethical GenAI-powered evaluation through global evaluation advocacy and partnerships. Leverage collaboration particularly in the Global South, share learning, and advocate for the importance of shaping GenAI-powered evaluation globally with UNFPA staff, partners, UNEG members and the broader evaluation community.
Marco Segone, Director, Independent Evaluation Office, UNFPA said:
“The advent of generative artificial intelligence (GenAI) has ushered in a new era of innovation and
transformation, offering immense potential to accelerate the Sustainable Development Goals (SDGs).
GenAI’s ability to analyse vast amounts of data, lowering barriers for obtaining automated responses,
and generate creative solutions may enable new approaches to addressing the pressing development
challenges. Recognising the potential of GenAI, the UNFPA Independent Evaluation Office (IEO) has
embarked on a journey of experimentation and innovation to maximize the potential benefits of this
new technology while minimizing potential risks, to strengthen the evaluation function by enhancing the
effectiveness, efficiency, and timeliness of evaluations.
This initiative aligns with the UNFPA strategic plan, evaluation policy, and evaluation strategy emphasizing
innovation, digitization, efficiency, and increased utilization of evaluative evidence for decision-making,
adaptation, and acceleration. The strategy – informed by a needs assessment – recognizes challenges and
risks in using GenAI in evaluation, offers strategic principles, and provides an implementation roadmap for optimizing the evaluation function.
This strategy will serve as a guiding framework for the innovative use of GenAI to power UNFPA evaluation function. It is also a valuable resource for the wider evaluation community to leverage GenAI to optimize evaluation, to accelerate progress towards the SDGs.”
Source: https://www.unfpa.org/sites/default/files/admin-resource/IEO__GenAI_Strategy.pdf, https://www.evalforward.org/resources/genai