James Wilsdon and Alex Rushforth
First half of session = hybrid; second half = onsite only
In the past decade, there have been a series of high profile efforts to tackle problems of overreliance on quantitative performance indicators and incentives, including the Declaration on Research Assessment (DORA), the Leiden Manifesto, the Metric Tide, and ongoing work by the Global Research Council. Most recently the Coalition on Advancing Research Assessment (CoARA) has emerged as a significant initiative, and now has more than 500 organisations committed to sharing ideas and evidence in support of responsible and inclusive assessment systems.
The rapid momentum of CoARA reflects a turn in these debates towards implementing, sharing and scaling solutions, within an expanding agenda of responsible research assessment (RRA), which includes more emphasis on healthy work cultures; research integrity; open scholarship; and principles of equality, diversity and inclusion.
At the same time, there is a need to underpin more normative, values-based agendas for reform of research assessment with robust and open evidence and debate as to what works, and which types of interventions are most effective, or may have unintended consequences. The task of aligning international capacity for meta-research, which has also grown over this period, with the priorities and agendas of RRA is still at a relatively early stage.
As a contribution to this task, this special session at STI 2023 will mark the launch of a new initiative – A Global Observatory of Responsible Research Assessment (AGORRA) – convened by the Research on Research Institute (RoRI).
AGORRA will aim to support and strengthen comparative analysis and progressive reform of national assessment frameworks, with a view to better aligning the needs of funders and other key actors in assessment systems with emerging meta-research capabilities. Through an international network of collaborators, AGORRA will run for an initial period of 5 years (2023-2028). Its specific aims are:
- To build and connect flexible research capacity that can be directed towards strengthening the evidence base for RRA and research assessment reform;
- To monitor ongoing developments in national and international frameworks and policies for research assessment, and support transfer and scaling of good ideas;
- To foster and facilitate a culture of analysis and experimentation among research actors engaged in the design, delivery and evaluation of assessment processes;
- To build evidence, awareness and engagement in the possibilities, pitfalls and choices being made with new methodologies (e.g. machine learning), indicators and ‘next generation metrics’;
To work, this initiative will require dialogue, engagement, and critical feedback from researcher, funder and broader communities. With this learning and improvement ethos in mind, we invite the STI audience to participate in this interactive session to discuss the promises, objectives, and potential pitfalls of this initiative.
The meeting will kick off first with a short overview of AGORRA by James Wilsdon and Alex Rushforth, followed by three flash pitches from AGORRA partners on the anticipated relevance of such an observatory to their respective national contexts: Steven Hill (Research England/UKRI) on the UK and future of the REF, Lin Zhang (Wuhan) on China, and Marta Wróblewska (University of Social Sciences and Humanities, Warsaw) on Poland.
The second half of the session will have an interactive format, with the audience invited into breakout groups, where they will explore the opportunities, challenges and priorities for a global observatory. Questions for discussion will include: Which interventions merit sustained attention by AGORRA? What would constitute appropriate evidence that a given intervention works? How can we improve learning and exchange across assessment systems? What might be the unintended consequences of scaling-up interventions across a range of organisational, regional and national funding contexts? These insights will then be fed back into the main group discussion.
The meeting will end by outlining future plans and identifying opportunities for the STI community to engage and contribute to AGORRA.
Research integrity – Retractions 1
