Kelly Woods, Senior Associate Publisher at F1000, recently participated in a panel examining current academic career assessment, research assessment reform, and the implications for open science.
Hosted as part of Eurodoc’s Annual Conference, we’ve rounded up some of the key takeaways of the panel, and you can find a full write-up of the panel here.
Linnéa Carlsson, Doctoral Committee Chairperson, Swedish National Union of Students
Linnéa opened the session by reflecting on her own experience of academic and research assessment, raising questions about the current status quo of publishing and its value for early career researchers (ECRs).
“When asked about my opinions on publishing my thesis by senior researchers and colleagues, I felt torn between those journals with the highest perceived reputation and alternatives. My colleagues would say “Of course you publish with those, why wouldn’t you?”.
But when you only have five years for your thesis, and it can take three to four years to complete the publication process for those ‘high-quality’ journals, it makes it difficult to choose which is best at this early stage of your career for career advancement.”
Kelly Woods, Senior Associate Publisher, F1000
Kelly discussed the intrinsic link between publishing and academic assessment historically, highlighting that traditional approaches have made assumptions about the quality of a person’s work based solely on metrics of the publication venue, for example, the Journal Impact Factor (JIF).
“Many groups have pointed out this flaw over the years, with metrics regularly being misused, encouraging us to avoid relying on these. Instead, many have agreed that the focus should be on things like article-level metrics – citations, views, downloads, and peer review reports.
However, a concrete alternative is yet to be established, and these venue-level metrics are still going into career decisions. Other key elements of career assessment are still being ignored, such as participating in peer review of research and sitting on editorial boards.
Peer review is a vital part of the research process but is currently overlooked in academic assessment and done behind closed doors. Opening this up can help to identify the time and energy spent on peer review, which can help to support the recognition of this in career assessment. Similarly, researchers are expected to conform to open science principles without much support, so providing guidance and more recognition and reward for open science participation is key.
It’s a time of great change for publishers and for career assessment. We know from our conversations with researchers that they really support the open model of Open Research Europe but are still cautious about publishing with alternative models due to a lack of support from institutions and colleagues. When researchers start to see the change that is outlined at a policy level, such as the recent Council Conclusions on Scholarly Publishing which support open peer review and giving recognition to those participating in peer review, we hope we will begin to see more adoption of alternative models.”
Sebastian Dahle, Vice President, Eurodoc
Sebastian highlighted that Eurodoc has long supported open science, underpinned by the belief that open science can enhance not only academia and wider society, but ECR experiences as well.
“Publishing research openly is a moral imperative, especially when funded by public money. I believe that those who fund the money should be able to see the outputs of what they have funded.
However, the question that is often asked is “Who will pay for open science?”. In actuality, open science can save us money, including saving resources by avoiding the repetition of previous research.
Open science is also key for ECRs and academic career assessment. It can help to generate more citations, and it can demonstrate the impact of your work now. As Linnéa mentioned, there is no point in having the impact in five years’ time from a high-quality journal if I need to apply for new jobs or funding in two years and need to demonstrate my work to do so.”
Jan-Ingvar Jönsson, Vice-Chancelor, Linköping University
Jan echoed the capacity of open science to change research assessment, but widened the discussion to look beyond publications when considering how we assess research.
“Assessing research is not just about what we publish. It should also be about providing open resources and educational materials, producing open source software and code, participating in academic citizenship, and influencing policymaking, to name a few academic skills. All these things can, and should, contribute to a change in the merit system, but there is no quick fix – we all need to contribute to the change in culture.
There is a big responsibility for academic leaders, but also funders, who can develop standards about how we assess research differently and provide incentives to encourage open science practices.
These changing metrics also need to reflect more than just individual assessment – they need to reflect how governments and institutions allocate funding and resources, for example. All the elements across the research chain need to be aligned otherwise it will break down.
As others have already said, there are lots of issues with current metrics, including:
- Encouraging competition over collaboration
- Celebrating prestige over novelty
- Promoting fixed research models over innovation
These matters pose a dilemma for ECRs. What are the incentives for ECRs to move towards openness? How can ECRs be involved without being the guinea pig?
There are also issues to consider for those assessing research. If each candidate presents achievements in different ways, how can you compare?
My advice to ECRs is to participate and influence open science whenever you can, for example, publishing thesis’ openly, having a realistic and honest review of the current model, and taking a lead in developing new models.
But it is imperative that it needs to be a collaboration across the whole scientific and academic community, and everyone needs to play their part.”
Karen Stroobants, Vice-Chair, Coalition for Advancing Research Assessment (CoARA)
Karen provided an overview of CoARA and how it came to fruition, including how it can help ECRs in research and academic career assessment.
“The development of CoARA was facilitated by the European Commission, with a recognition of the momentum research assessment reform was gaining in the research community and a desire to place the topic higher on the political agenda.
It began with stakeholder engagement to produce the Agreement on Reforming Research Assessment (ARRA), through a co-creation process resulting in a common direction to move forward the reform of research assessment, including recognition of wider open science activities.
The Coalition is bringing together all those willing to engage in the reform, underpinned by the four core commitments in the Agreement:
- Recognise the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research
- Base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators
- Abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index
- Avoid the use of rankings of research organizations in research assessment
The Coalition brings stakeholders together at different levels who all adhereto the commitments – as Jan said, we need national, funder, institutional, and individual stakeholders to interact and align.
The aim is not to just swap one narrow model for another. Instead, we need a diverse number of research assessment tools for different use cases and organization types, while ensuring alignment as best as possible to enable effective comparisons and not impede researcher mobility between roles and institutions.
However, this is not just needed at a European level – we require it worldwide to ensure effective collaboration and researcher mobility.
The agreement itself stresses the importance of ECR involvement in changes to processes, raising awareness among ECRs of new measures, and ensuring that there won’t be any negative impacts on young researchers from these changes.
I will join Jan in saying that ECR support is crucial to progress the reform – they will be the ones able to identify any issues that arise from changes, and they will be the future leaders, so it’s imperative they shape something that works for them.”
Research assessment reform and Open Research Europe
Open Research Europe already implements many practices that support research assessment reform, such as:
- No Journal Impact Factor, instead promoting the responsible use of individual article indicators.
- Faster publishing times to demonstrate the impact of work without lengthy delays, due to our post-publication peer review process
- An open data policy that facilitates the reproducibility of research.
- 14 article types supported, including publication of confirmatory, null, and negative results, and dedicated Data and Software article types for diverse outputs.
- Opportunities for increased credit for additional activities, such as Advisory Board roles and peer reviewing (each peer review report can be cited independently from the article).
Interested in the full panel round-up? Find all the Q&A responses here.
