Introduction
One of the scientific research objectives is to test theories, typically by accumulating knowledge derived from individual studies. This process is contingent on the reliability of previous findings that serve as the foundation for subsequent research. One of the essential prerequisites for trustworthy research findings is reproducibility, which entails obtaining the same results when rerunning (parts of) projects using the same design and data. Unfortunately, research findings frequently cannot be reproduced due to, for example, inadequate information provided to rerun the experiment or analysis (Alsheikh-Ali et al., 2011; Stodden et al., 2018). The EU report on the reproducibility of research results in EU framework programmes (European Commission, Directorate-General for Research and Innovation, 2022) shows that many researchers rarely indicate to share their data, analysis code, or materials. This is problematic not only for the reproducibility of experiments but also for their ability to be replicated in other samples. Ensuring the reproducibility and replicability of scientific results is critical to the rigor and quality of research. When they are ignored, there is a risk that inaccurate, biased, or spurious results can gain undue attention in the literature leading to substantial research waste and flawed decision-making. Strategies to increase reproducibility and replicability have primarily focused on improving research transparency through various open science practices. These aim to ensure that the research process is documented and widely accessible so that it can be checked, critiqued, re-used, and built upon in future research. Ongoing initiatives, such as the ReproducibiliTEA network, aim to facilitate the implementation of these practices in the research process. However, the extent to which these and other practices and interventions have been empirically investigated, and their actual impact on reproducibility and replicability, are unclear. We, therefore, aim to understand the evidence base for interventions for increasing the reproducibility and replicability of research and the documented barriers and facilitators in the process of creating more reproducible and replicable research using scoping review and evidence mapping methodology.
Methods
This protocol was developed prior to the search and will be pre-registered on OSF under DOI 10.17605/OSF.IO/D65YS after including the final search strings. Title and abstract screening started in June 2023 based on a finished protocol. A well-constructed search strategy is the core of the systematic review. Due to the complexity of our search objectives, we revised the search string throughout the T&A screening in order to improve its quality. The final search string will be published with this protocol. The final search string for Medline can be found in the section describing the search strategy. Potential changes to the protocol during the research process will be reported transparently. All additions to the protocol, such as data extraction sheets and data analysis plans, will be added to the OSF folder of this project under DOI 10.17605/OSF.IO/7EF5H whenever they are ready. The draft protocol has been reviewed by all authors. The protocol follows the Preferred Reporting Items for Systematic Reviews and Meta- Analyses: extension for Scoping Reviews (PRISMA-ScR) guidelines (Tricco et al., 2018).
Design
Scoping review methodology will be applied since overviews regarding the evidence of the effect of interventions on reproducibility and replicability still need to be provided. We want to create an overview of the current evidence by systematically mapping available literature using the EPPI Reviewer software (EPPI Centre Software). By doing so, we aim to identify knowledge gaps and gain an overview of the characteristics of the current literature.
Objectives
Our primary objective is to evaluate which interventions, such as open science practices, have been investigated for their effectiveness in improving reproducibility and replicability in science. These interventions can be applied on various levels (researchers, institutes, funders, publishers, editors, etc.). The search strategy will focus on this objective.
The secondary objective is to investigate which drivers and barriers to the implementation and effectiveness of these interventions have been identified in these interventional studies.
Definitions
For the purpose of this review, we embrace the definitions of replicability and reproducibility from Nosek et al. (2022) and European Commission's scoping and final report on reproducibility in research (European Commission, Directorate-General for Research and Innovation, 2020 and European Commission, Directorate-General for Research and Innovation, 2022). Following these definitions, we created a list of interventions and outcomes that will be investigated, which can be found in Figure 1 and Figure 2. By adopting this list, we acknowledge that it might not be exhaustive as improving reproducibility in science is a continuous exercise, and there are many maturing ideas tackling the challenge (Munafò et al., 2017).
Figure 1. Outcomes.
Figure 2. Interventions.
Eligibility criteria
We will include studies evaluating the effectiveness of an intervention in promoting the reproducibility or replicability of scientific methods and findings and those additionally testing drivers and barriers of the process. Studies will be included independent of the specific approach applied to measure the outcome variable(s), such as using proxies or direct indicators of the outcomes.
In addition to studies that explicitly evaluate interventions for reproducibility/replicability, we will also consider studies that assess the effectiveness of interventions related to practices commonly claimed within the literature to support reproducibility/reproducibility to be in the scope of this review. The complete list of such proxies is included in Figure 1 and is based upon two criteria: the project team’s synthesis of key, widely cited prescriptive texts that provide a conceptual framework for the selection made (Munafò et al., 2017; Stodden et al., 2016; European Commission, Directorate-General for Research and Innovation, 2020 and European Commission, Directorate-General for Research and Innovation, 2022) and the team’s evaluation which proxies would be measurable.
Various study designs will be considered for inclusion in this research. To evaluate an intervention, comparisons must be made either between or within subjects. Studies that do not have a comparator will be included only if the intervention is explicitly stated and the outcome variable measures reproducibility or replicability (for example, the prevalence of data sharing in journals with open science policies). These studies are included since they provide relevant information when compared with other studies (e.g., the prevalence of data sharing in journals without open science policy). Studies only investigating the prevalence of certain practices, such as data-sharing, are not included. Reviews will be included to apply the snowballing methodology to identify relevant literature, and since they might contain additional relevant information that is retrieved by the comparison and summary of primary studies. Reviews will be summarized separately to avoid duplication of primary studies in the evidence map. Furthermore, we include studies that investigate facilitators and barriers in addition to evaluating the effectiveness of interventions on reproducibility and replicability (e.g., as moderator). When only drivers and barriers are assessed, for example, in the form of a survey or similar describing the opinions of multiple individuals without testing any intervention, these studies will be marked as such in the full-text screening phase. Including these articles in the analysis is beyond this project's scope. However, these papers are essential for understanding researchers' practices and decisions and compiling them might serve as a starting point for further projects. Article types such as position papers, study protocols, and other literature without primary data will not be considered. Additionally, we will search for and exclude retracted articles.
Participants: Researchers, institutes, funders, publishers, editors etc. in any field of research.
Interventions: Any intervention that aims to improve the reproducibility and replicability of science. These can be open science practices, such as the pre-registration of studies, open access publishing, data and code sharing or interventions to promote these open science practices, such as journal editorial or institutional policies, and implementation of guidelines or codes of conduct.
Comparator: Any comparator, including the absence of a comparator (e.g., pre-post comparison).
Outcomes (qualitative and quantitative): Reproducibility, replicability, and their proxies.
Article types: Articles, review articles and early access papers will be included.
Study types: Qualitative and quantitative interventional studies (e.g., pre-post or experimental designs).
Search strategies
Electronic searches. At first, we will identify key papers on the topic across various domains. This will be done by direct consultation with colleagues and reproducibility and open science experts for literature recommendations to fill any remaining gaps. The key papers will be expanded by citation coupling using Connected Papers. The team will screen and discuss the new list, including an information specialist. The resulting set of documents will both inform the building of a search query and validate the proposed search. The articles will be added to a publicly available Zotero folder
Additionally, the team will create a list of interventions and outcomes of interest (see Figure 1 and Figure 2). This list will be informed by the team’s knowledge of the field, colleagues' consultation, and the key papers' screening. The search terms will be defined based on the list and key papers. Before searching, the search strategy will be checked by an external information specialist.
We will systematically search Medline, Embase, Web of Science, PsycINFO, Scopus, CAB Direct, Agris, PubAg, AGRICOLA and Eric. The proposed search will initially be developed for Medline and subsequently be translated to all databases. The search string for Medline can be found at the end of this section. No date or language restriction will be used. Grey literature will be searched separately by scanning websites of major policy actors (such as EC, Science Europe, EUA, NSF) for relevant interventional studies. After identifying relevant references, they will be used for reference and citation checking to determine potentially missed studies. The results from the searches will be collected in EndNote 20. Duplicate references will be removed with an in-built deduplication function of EndNote.
Medline Search string. (((data or code or workflow or practices or materials or notebook) adj2 (open or share or shared or sharing or preservation or stewardship)) or "open science" or ((computational or data or open or research or conclusion* or inferential or analytic or conceptual or direct or exact or statistical) adj3 (reproducib* or replicability or replicable)) or (research adj5 (transparen* or credib*))).ti,ab. or (reporting adj3 guideline*).ti.
(Clinical study/ or Case control study/ or Family study/ or Longitudinal study/ or Retrospective study/ or Prospective study/ or Cohort analysis/ or Comparative Study/ or (Cohort adj (study or studies)).mp. or (Case control adj (study or studies)).tw. or (follow up adj (study or studies)).tw. or (observational adj (study or studies)).tw. or (epidemiologic$ adj (study or studies)).tw. or (cross sectional adj (study or studies)).tw. or (comparative adj stud*).mp. or (("randomized controlled trial" or "controlled clinical trial" or "multicenter study" or "pragmatic clinical trial").pt. or non- randomized controlled trials as topic/ or interrupted time series analysis/ or controlled before-after studies/ or random*.ti,ab. or groups.ab. or (trial or multicenter or "multi center" or multicentre or "multi centre").ti. or (intervention? or effect? or impact? or controlled or control group? or (before adj12 after) or (pre adj5 post) or ((pretest or "pre test") and (posttest or "post test")) or quasiexperiment* or quasi experiment* or pseudo-experiment* or pseudoexperiment* or evaluat* or "time series" or time-point? or "repeated measur*" or ((experimental or empirical or qualitative) adj5 (study or studies))).ti,ab.)) not ((news or comment or editorial).pt. or comment on.cm.)
Selection process
The title and abstract of all records identified through the respective search strategies will be screened for potential eligibility in duplicate independently. When there are disagreements, a third reviewer will be involved to resolve discrepancies in the selection of articles. The full-text manuscripts belonging to the titles and abstracts considered potentially eligible will then be read and selected for inclusion in duplicate independently. In this phase as well, disagreements will be solved by a third author.
Data-extraction
The data extraction sheet, including extraction instructions and terminology definitions, will be developed in cooperation with the whole team. After extracting the first ten papers, the extraction sheet will be piloted and potentially adapted. Data extraction will be conducted in duplicate by two or more reviewers independently. Disagreements will be solved through discussion and additional consultation of a third reviewer. Data will be extracted from study documents and recorded in EPPI Reviewer across the items listed below. In case of insufficient data, we will attempt to contact the document’s authors for additional information. The following data will be extracted (if applicable):
a. Description of the publication
b. Type of publication
c. Study design (in case the publication was a literature review, this item lists the study designs included in the review)
d. Description of the sample
- Sample
- Location/context
e. Interventions
- Classification according to list of interventions in appendix A
- Stringency (mandatory, optional)
- Implemented by
- Target population
Stage of target process (before, during, after study conduct, after publication)
f. Outcome variables
g. Drivers and barriers
h. Results
- Answer to research question
- Primary statistical results (effect size, p-value, CI, etc.)
- Descriptives (means, percentages, etc.)
- Meta-analytic results
- Qualitative results
- Additional results (e.g., drivers/barriers)
Critical appraisal of individual sources of evidence
As this is a scoping review, we will not assess the risk of bias of each individual study included. We will, however, check for each article what the design was and whether a comparison was included (see data-extraction items).
Data synthesis and analysis
All included studies will be summarized in a narrative synthesis.
The evidence will be mapped according to the intervention (rows) and reproducibility/replicability (columns) using the EPPI Reviewer software (EPPI Centre Software). Additionally, an adapted version of the categorization, as described by Davidson et al. (2022), will be applied to cluster different kinds of interventions.
The outcomes of reviews will not be included as a row in the evidence map but described separately. Evidence mapping will produce an interactive, searchable database of relevant studies, a list of knowledge gaps, and visualizations such as evidence atlases, heat maps, or descriptive plots. If possible, this will be done semi-automatically as well (using EPPI-reviewer, for example). These will aid in identifying gaps in the evidence and ways to address them. The exact layout of these figures will be determined after we have piloted the data extraction sheet.
Study status
We are currently at the stage of title and abstract screening.
Anticipated end of the study
The anticipated end date of the study is September 2023.
Ethics and consent
Ethical approval and consent were not required.
