An Agreement on reforming research assessment has recently been published, setting a commonly agreed direction for changes in assessment practices for research, researchers, and research performing organisations, with the overarching goal to maximise the quality and impact of the research.
The Agreement is the result of a co-creation exercise facilitated by the European Commission and involving over 350 organisations from 40 countries. These include public and private research funders, universities, research centres, research institutes and research infrastructures, and associations and alliances thereof, national and regional authorities, accreditation and evaluation agencies, learned societies and associations of researchers, and other relevant organisations, representing a broad diversity of views and perspectives.
What is research assessment?
Research assessment evaluates researchers’ work to select them for funding, recruitment, and promotion opportunities. It is fundamental to the research enterprise as it informs decisions to invest finite resources, evaluate return on investment, and support the best science and scientists.
How is research currently assessed?
Research assessment often relies on narrow indicators to evaluate quality and the resulting impacts of research such as the number of publications and the prestige of the venues where these are published.
Some of these indicators, like the Journal Impact Factor and h-index, have several well-documented deficiencies as tools for research assessment. Among other issues, they fail to recognise the diverse contributions of researchers and they are field dependent, which may lead to objectionable practices.
Several initiatives, such as DORA, Leiden Manifesto, and Hong Kong Principles, agree on the inadequacy of using quantitative indicators as the only way to measure quality, and have offered alternatives as to how best to reform research assessment.
Although these proposals have triggered important conversations, and many researchers, funders, and research institutions have committed to their principles, progress has remained slow and inconsistent, both across Europe and worldwide.
How will the Agreement improve the current situation?
The Agreement proposes a reformed research assessment system based on principles and commitments that could be agreed between organizations involved in research assessment, as well as a concrete timeframe for implementation, while fully respecting the autonomy of organizations.
In summary, it proposes four core commitments that call for action to:
- Recognize the diversity of contributions to, and careers in research.
- Base research assessment primarily on qualitative evaluation supported by responsible use of quantitative indicators.
- Abandon the inappropriate uses in research assessment of journal and publication-based metrics.
- Avoid the use of rankings of research organizations in research assessment.
It also contains six supporting commitments to pilot and enable the move towards new criteria, tools, and processes for research assessment, to facilitate mutual learning and to communicate progress made, and to ensure that new approaches are evidence informed.
What does this Agreement mean for Open Research Europe?
This is an exciting announcement for Open Research Europe, as the Agreement would contribute to mainstream practices that support robustness, openness and transparency of research and the research process. Open Research Europe already implements many of these practices, such as:
- No Journal Impact Factor, instead promoting the responsible use of individual article indicators.
- Early sharing of results with open post-publication peer-review as well as an open data policy where research data supporting articles is deposited in trusted repositories, facilitating reproducibility of research.
- 14 article types accepted across all subject areas, including publication of confirmatory, null, and negative results.
- Education, training, and support for researchers for peer-review and open research practices.
- Opportunities for increased credit for additional activities undertaken by researchers, such as Advisory Board roles and peer reviewing (each peer-review can be cited independently from the article).
This Agreement will be open for signatures in September 2022, creating a coalition of institutions who will work on the reform of research assessment. To learn more about the Agreement, visit Reforming research assessment: The Agreement is now final | European Commission (europa.eu)
