Introduction
In recent decades, research integrity (RI) and responsible conduct of research (RCR) have developed into a fundamental cross-cutting issue in research as well as in the training of (future) researchers (Abdi et al., 2021; Gerber et al., 2020).
Nevertheless, there is still no substantial evidence to show what works and what does not work in teaching and learning RI/RCR (Antes et al., 2009; Godecharle et al., 2013; Marusic et al., 2016; Steneck, 2013).
We acknowledge that RI and RCR tend to overlap and blur into each other in practice, however we align with the European Code of Conduct for Research Integrity (2017). As such we describe RI for this project as a commitment to professional, legal, and ethical responsibilities, values, and principles to self-regulate research (cf. ECoC, 2017, 3ff.). RCR on the other hand is “simply conducting research in ways that fulfil the professional responsibilities of researchers, as defined by their professional organizations, the institutions for which they work and, when relevant, the government and public” (Steneck, 2006, 55), in particular “RCR education can help science take care of itself” (Roth, 2002).
RI describes therefore the framework, such as the ECoC (2017) while RCR concentrates on conducting research that is in line with RI. Since the Evaluation of Path2Integrity focusses on the question, if the practice and justification for this is in line with RI, we use the term RCR training to refer to training in the field of RI and RCR.
One of the main challenges in fostering a research integrity culture through training is that there is a lack of substantial evidence to show what works and what doesn’t in teaching and learning RCR. One reason for the lack of evidence is that standardised instruments to evaluate the success of such interventions are rare, particularly for quantitative assessment. Most studies assessing successes of RCR training have used qualitative evaluation tools (e.g. Aubert Bonn & Pinxten, 2021; Hyytinen & Löfström, 2017; Sørensen et al., 2021).
To overcome this lacuna, we designed the so-called P2I questionnaire. This instrument concentrates on practices and justifications in line with RI. It assesses the understanding of and argumentation for research integrity as conceptualised in the European Code of Conduct for Research Integrity (ECoC, 2017). We designed the P2I questionnaire to assess the learning successes of the P2I training programme (Prieß-Buchheit, 2020b) and to compare the test results in a pre-post test design from secondary school students older than 16 up to early career researchers.
The following shows the questionnaire’s development process in three steps. After describing the status quo and gaps in evaluating RCR training successes first, this article illustrates how we designed the P2I questionnaire using practices and justifications in line (and not in line) with research integrity in a second step. In a third step, the P2I questionnaire is pre-validated and revised. Developing such a questionnaire is a first step towards evaluating RCR training’s successes in a standardised way. By making each design transparent, we open a discourse to reach a standardised instrument at the end, quantitatively assessing learning successes in RCR training.
In particular, we acknowledge that different designs of assessment instruments significantly impact what they evaluate. We took that into account and elaborated on a specific design in the development process of the questionnaire. Whereas traditional multiple-choice questionnaire often assess knowledge on one tier, a multiple-tier structure can give insights into students’ justifications. That is why we decided to use a multiple tier structure for the P2I questionnaire
We describe the development process of the questionnaire and its specific items as transparent as possible, to establish a comprehensible, valid, and reliable instrument. We collected answers in line (and not in line) with research integrity for different items as one key element in designing the P2I questionnaire. Furthermore, students and experts proofed each item in the pre-validation process. Some items were ambiguous and have been revised. The result is a pre-validated questionnaire. We publish this (early) design procedure to invite other experts to comment and improve the P2I questionnaire. Hence, this article does not yet present the final validation nor evaluation results.
Status quo and gaps in evaluating successes of RCR training
As previously mentioned, there are different qualitative instruments used for measuring the successes of RI/RCR training. For example, Rissanen & Löfström (2014) use vignettes, Vehviläinen et al. (2018) conduct qualitative semi-structured interviews, Poom-Valickis & Löfström (2019) administer interviews, Sarauw et al. (2019) use observations, and Berling et al. (2019) combine toolbox discussions with self-assessment.
Quantitative instruments to assess RCR training's successes remain rare, however attempts to implement quantitative instruments derive mainly from ethics education. This is because of overlapping training content and similar learning methods. Three of the most frequently used instruments are the Defining Issues Test (DIT) (Rest, 1979), its revised version, the DIT-2 (Rest et al., 1999), and the sensemaking approach (Mumford et al., 2006).
While the DIT aims to assess cognitive moral reasoning schemes, which focus on dilemma situations, the sensemaking approach assesses specific metacognitive reasoning strategies (Mumford et al., 2008). Although both aspects are demonstrably relevant in RCR training, they are derived from ethics education and do not capture the RCR training core theme as described in the ECoC (2017): RCR training aims to enable participants to become a part of the research community, including “the principles of research, … the criteria for proper research behaviour, [and] … the quality and robustness of research” (ECoC, 3).
What does that mean in particular? There are different ways to transform this overall aim to become part of the research community – into a training. Prieß-Buchheit et al. (2020a) describe that Path2Integrity’s approach to training is to learn to argue in favour of RI. The theory presupposes that a responsible (future) researcher can argue in favour of practices in line with RI (Prieß-Buchheit et al., 2020a). Hence, after a RCR training, participants should be able to choose practices in line with RI over other practices not in line with RI and should be able to justify their choice. Learning to argue in favour of RI entails different topics such as, how to cite correctly, how to handle research subjects, and hoe to work in collaborative teams etc. (see e.g. Hermeking & Prieß-Buchheit, 2022).
However, a common research integrity challenge is, that one practice can simultaneously be both in line and not in line with RI. For example, the practice to publish many articles can on the one hand, be driven by the purpose to get promoted and therefore have enough money to feed one’s family; and on the other hand, could be driven by the purpose to timely inform others about one’s findings. The former argument could lead to a salami publication tactic, whereas the latter does not run the same risk. How somebody justify their practices is crucial.
Another instrument, called the Professional Decision Making in Research (DuBois et al., 2016) was developed based on the Ethical Decision Making (Mumford et al., 2006). This assessment instrument focusses (not on justifications but) on decisions and “ethical problems that include factors such as incomplete knowledge, power discrepancies, and urgency – factors that may interfere with ethical decision making” (DuBois et al., 2016). Also, the Academic Motivation and Integrity Survey, as for example presented in Stephens et al. (2021), and the Academic Integrity Self-Evaluation Tools, as reported in Gaižauskaitė et al. (2020) cover the field of research integrity. They all focus on ethical decisions and behaviour.
In distinction to these tools – which concentrate on ethical decisions and behaviour – our approach assesses, if (future) researchers argue in favour of research integrity. With this approach we ensure that the assessment reveals what practice in line with RI people chose in the research community and how they justify this practice. Participants show how responsible they are by choosing one option on each item representing the ECoC’s topics of RI.
Until now, there has not been an established standardised instrument that builds on a common reference document such as the ECoC (2017) and covers peoples’ argumentation in favour of RI. Nevertheless, some evaluations use a promising structure to assess how people argue in favour of RI. In 2007 Chou et al., applied the so-called two-tier structure – a standard multiple choice question supplemented by a second tier in which the choice is to be justified – to a research integrity-related topic. They assessed students' justifications of cyber copyright laws. Also, Sun (2009) examined the paraphrasing strategies with a two-tier approach, while Pan & Chou (2015) used this structure to assess students' misunderstandings about ethics and behaviour. These approaches demonstrated that a two-tier test seems to be a good fit for a RCR questionnaire concentrating on practices and justifications (not) in line with RI.
Methods
The aim of the P2I questionnaire is to quantify a RCR training's success by looking at the number of answers from the respondents in line with the ECoC (2017). The instrument aims to assess what practice testees would choose in research settings and how they argue in favour of these practices. Practice in this sense means a concrete action, what to do to follow good research practice. The questionnaire also should observe how confident testees are in their answers. This helps to identify whether an answer was given due to strong believe or due to uncertainty. Trainers should be able to see both, how confident testees choose practices in line or not in line with RI and how confident they justify this practice to be in line with RI. This means that we focussed and limited the P2I questionnaire to what practices testees choose and how they argue for this practice in line with RI.
In the following we describe the development of the multi-layered-tier P2I questionnaire. As shown in Figure 1, we developed the questionnaire by following Chandrasegaran, Treagust, and Mocerino’s (2007) approach and design, and pre-validated the P2I questionnaire in three steps.
Figure 1. Scheme of development of the P2I questionnaire adapted with permission of The Royal Society of Chemistry, from Chandrasegaran et al. (2007); permission conveyed through Copyright Clearance Center, Inc.
Figure 1 describes the process from step one, defining RCR content to step two, identifying practices and justifications in line (and not in line) with research integrity to result in step three, developing the structure and items to evaluate the RCR training successes.
Step one “Defining the content” is twofold: first we identified RI topics which seem to be relevant in the research environment as well as for the target groups. Then we adapted these topics in regard to different target groups in RCR training. In step two we defined the answers for the first tier (practices in line and not in line with RI) and the third tier (justifications in line and not in line with RI) based on the specific RCR content that was identified in step one. Step three outlines the development of the structure of each item, starting with the pre-validation of each item and refining the items after the first tests runs and with the help of experts.
Ethics statement
All procedures performed in developing the questionnaire, as well as in the questionnaire were approved by the institutional research committee (Central Ethics Committee of the University of Kiel) approval number: ZEK-10/20.
Written informed consent for publication of data was obtained from all subjects before they conducted the P2I questionnaire. All testees proactively clicked on the ‘next’ button in the online questionnaire to proceed with the voluntary test and checked a mark that they voluntarily give data for this study.
Step one: Defining the content
Designing an accurate, reliable, and valid questionnaire requires considerable knowledge, skill, and experience in question-wording and questionnaire structure. Furthermore, our study requires that we then utilise this questionnaire to evaluate RCR training success which additionally requires critical knowledge on the subject area.
To identify research integrity topics in RCR training we followed Roth (2002), who says: "Since science is self-policing, it may be tempting to think that the scientific community can handle any matters of responsibility by its own methods. This is already rebutted by the creation of regulations to govern scientific research due to past failures of the scientific community to minimise and mitigate misconduct by some scientists. Moreover, RCR education raises issues for scientists to promote reflection and consciousness of their roles as members of the scientific community. Thus, RCR education can help science take care of itself".
An aim of this study is to quantify the success of RCR training. To do so, we have conceptualised our understanding of RCR as follows: First, we adopted an approach from ethics (education), which concentrates on establishing and explaining standards or principles of moral behaviour – which can be differentiated into values, norms, and virtues (Peels et al., 2019) – within a community. Additionally, we focussed on integrity, emphasising the practices of agents who comply with these ethical standards or principles (Löfström & Pyhältö, 2019). We narrowed the questionnaire’s focus to be in line with Shaw’s (2019) understanding that "research integrity lies in consistency with external rules" (p. 1087) and aimed to assess what practices testees choose in research settings and how they argue in favour of these practices.
To identify what content should be included in the questionnaire, we clarified what common ground we expect all RCR trainings to have. In a nutshell, Jordan (2013, 245–6) differentiates ethics from integrity. She describes ethics as "the standards of moral behaviors expected of autonomous humans living in a community, often through reference to a coherent system of thought or an ethical theory (e.g., deontology). [In opposition to that,] integrity describes how an individual or institution brings their moral positions and behaviors together in a logically coherent position that is brought into daily practice." This corresponds with Steneck (2006, 56), who describes research integrity as the "quality of possessing and steadfastly adhering to high moral principles and professional standards, as outlined by professional organisations, research institutions and, when relevant, the government and public". Following Jordan's statements above, learning RCR is not about blindly following rules (history has taught society many times the consequences of such attempt). Instead, learning RCR is about following rules due to a discourse based on rational arguments that legitimises all statements that claim normative validity and must be agreed upon by individuals (Habermas, 1990 and Gethmann, 2010). In Europe we have the ECoC (2017) as a common framework, displaying such a discourse. Such a perspective emphasises the individual's personal responsibility regarding research integrity (Frankel et al., 2015), which is (only) compatible with RI/RCR training but not sufficient for fostering a culture of research integrity (Valkenburg et al., 2021).
In identifying good research practices, we concentrated on the ECoC (2017), which names the following RI topics: Research Environment, Research Procedures, Safeguards, Collaborative Working, Publication and Dissemination, Data Management, Mentoring, and Reviewing, Evaluating and Editing. Our aim was to assess training from secondary school students to early career researchers. According to the European Framework for Research Careers, Reviewing, Evaluating and Editing are getting relevant for established researchers (R3) (European Commission, 2011, 10) Therefore, we decided to include all the fields from the ECoC (2017) mentioned above, except the field Reviewing, Evaluating and Editing.
As shown in Table 1, we included the fields Research Environment, Research Procedures, Safeguards, Collaborative Working, Publication and Dissemination, Data Management, as well as Mentoring and designed items. We created two versions of the questionnaire. The S and the M / Y versions adapt, transform, and reduce the above-described topics to two target groups by choosing concrete content from these topics. The two versions account for academic discrepancies between the target groups. Namely that secondary school students and bachelor students do not have an academic degree yet (S version) (Zollitsch et al., 2020b), whereas master students and early career researchers hold an academic degree (M / Y version) (Zollitsch et al., 2020a).
Table 1. Content of the P2I questionnaire (referring to the European Code of Conduct).
| Version S | Version M / Y |
|---|
| Research Environment | X | X |
| Research Procedure | X | X |
| Safeguards | X | X |
| Collaborative Working | X | X |
| Publication and Dissemination | X | X |
| Mentoring | | X |
| Data Management | | X |
As a result, the P2I questionnaire contains two versions with several items. The items in each version adapt to pre-collegiate and collegiate learning circumstances. The testees see the items and each response possibility in random order to ensure there is no bias within the items. In this way, we reduce the possibility that there is a dependence between items, which might allow testees to answer an item more easily because of the previous item that they already finished.
Step two: Identification of practices and justifications which are and which are not in line with RI rules and principles
To identify which practices are in line and which practices are not in line with RI, we first designed different scenarios relating to the ECoC (2017) topics. To find suitable multiple choice answers and distractors to the designed scenarios, we then conducted group discussions with both students and researchers. In total, we conducted six group discussions with students majoring in education as well as with an interdisciplinary research group at Kiel University in Germany The persons in the group discussions were not randomised; participation was voluntary. Participants were approached personally since this procedure involved a considerable investment of time for the participants and no compensation was offered for their participation. To make the procedure as efficient as possible students with high intrinsic motivation in research, and a lack of any particular connection to RCR were asked to help improve the questionnaire.
We presented the first case scenarios to each focus group and asked them what one should do to act in line with RI. We sought open and free-response questions. With this procedure, we collected attractive practices in line and not in line with RI for two different target groups: a) secondary school students and undergraduates and b) master students and (early career) researchers. Given that our focus group participants were both students and researchers, we were able to collect attractive practices from their living environment.
In a second step, we analysed common justifications to find out which justifications are not in line with RI and are more common in a specific test group, culture, gender, or age. We designed a draft questionnaire with seven items and open-ended questions to identify justifications and justification patterns for the aforementioned practices (Draft of the P2I questionnaire, Version 1.0) (Zollitsch et al., 2019a).
A total of 35 participants, from both target groups (aged 18 to 50) filled out the draft questionnaire. The participants were, mainly from the social sciences and were equally gender-mixed from Kiel University in Germany, they were approached personally, participation was voluntary, and the group was not randomised. In order to reach the participants, the lecturers of the Department of Education at Kiel University were asked to have students fill out the questionnaire within their courses. In the end, the participants came from six separate courses with four different lecturers. The participants answered the draft questionnaire in a pen-and-paper version.
The following displays some original justification examples:1
- Research means being creative and not always sticking to rules.
- In research, one should think outside the box and break away from the rules to discover new things.
- The regulations are often written in a complicated way and do not provide a good reference point for writing a paper.
- One should not refer purely to regulations because there is little practice associated with it.
We analysed all 232 free text responses carefully and elaborated repeating justifications by doing a qualitative content analysis, as described in Mayring (2000). The development of inductive categories revealed justification patterns from the everyday life of the target groups that we used. The answers were reduced to their content-bearing components and comparable answers were combined in different patterns. Finally, a counter check with the original answers confirmed the final structure. As shown in Table 2, the answers from the participants led to nine different patterns that contain different justifications as well as one justification pattern according to the ECoC.
Table 2. Justification patterns (not) in line with RI.
| Justification patterns | Answers |
|---|
Scientific common sense
(according to the ECoC) | It ensures reliable research results. |
| Hierarchy | The supervisor said X has to do it that way. |
| Structural conditions | The structural conditions have made this scientific practice necessary. |
| Individual Benefits | This scientific practice will support X’s career and get him/her the next Nobel prize. |
| Community benefits | Only with this scientific practice can the highest benefit for the community be ensured. |
| Equal treatment of all | The same requirements should apply to everyone. |
| Duty | Good researchers must do this. |
| The others | As long as others do this, X can do it. |
| Quantitative majority decisions | Since the majority does so, X must adapt his/her scientific practice accordingly. |
| Rejection of binding codes | Due to the complexity and variety of research, there can be no binding codes and
regulations. |
We selected the justification pattern according to the ECoC as well as three justification distractors for each scenario. We reformulated the distractors as little as possible, as we wanted to stay close to the typical answer while also ensuring connectivity to the scenario of the item.
In the following, we use the term scientific common sense (SCS) when the statement is in line with the RI rules and principles from the ECoC. We combine different statements, which are in line with RI, in a justification pattern called the ‘scientific common sense pattern’. Statements from this pattern are distinguishable from other statements, which are not in line with the scientific common sense, such as the rejection of binding codes or individual benefits (as presented in Table 2). We take into account that statements other than our predefined SCS statements could also be correct and expected in specific settings other than our RI-setting.
Step three: Development of the structure and items
To assess the success of RCR training, we choose a so called multi-layer-tier structure for the questionnaire, specifically we used models from the co-called tier-test designs. Treagust (1988) designed the original structure of the two-tier test as an extension of classic multiple-choice tests to determine not only what students know but also how students understand a specific topic. This two-tier approach can help differentiate between multiple choice answers which are right by chance and which are right because there is an understanding of the concepts. Treagust's original two-tier test is a two part (a.k.a. tier) multiple choice series. It starts with a multiple-choice question that the testees respond to in the first tier; this extends to a second-tier where the testees reason their response from the first tier. He based his original design on Tamir's (1971) approach in which distractors were derived from existing misconceptions of the testees – this would make them more attractive. The two-tier test is an extension of classical multiple-choice tests and helps to determine not only what students know but also what rationale underpins that answer.
Such a multi-layered tier test provides quantitative data from both tiers and provides insights into the first tier's quality (Yang & Lin, 2015). This design suits the needs of this study as it allows statements about the testees' choice of appropriate practices in RCR and the justifications for these practices. Furthermore, it identifies frequently occurring and stable rationales. As Hestenes & Halloun (1995) or Yan & Subramaniam (2018) show, the two-tier test is able to show what action testees would suggest in a situation and, at the same time, why they would do that. Following this idea, we designed the P2I questionnaire as a multi-layered tier test, assessing the testees' choices of research practices and justifications, as Tsui & Treagust (2010), Chandrasegaran et al. (2007), or Treagust (1988) outlined.
First pilot test and revision
For the first pilot run, 11 students conducted the two-layered P2I questionnaire. All 11 students were from the educational field (six bachelor students and five master students) at Kiel University in Germany, but were from different courses. They were given a a pre-version (Pre-Version of the P2I questionnaire, Version 1.0) (Zollitsch et al., 2019b), derived from the draft questionnaire (Zollitsch et al., 2019a). The group was not randomised; the participation was voluntary. In order to acquire participants as efficiently as possible, the lecturers who had already agreed to support the study before were approached again. To avoid distorting the results, we only selected courses in which participants had not already completed the test. This left two courses where the questionnaire could be distributed. The participants answered six items: two items for the category Research Procedures, two items for the category Collaborative Working and two items for the category Publication and Dissemination.
Their results in Table 3 show that at least six out of 11 participants answered in line with RI in both tiers in all six items. Item cd9 received the same number of answers in line with RI in the first tier and the second tier. All other five items show that even if the participants answered in the first tier in line with RI, they did not always choose the justification in line with RI.
Table 3. Results of the first test.
| Scientific practice fields | Research procedure | Collaborative
working | Publication and
dissemination |
|---|
| Item | cp3 | cp4 | cc1 | cc5 | cd2 | cd9 |
| Tier 1 practice in line with RI (%) | 8 (72.73) | 10 (90.91) | 7 (63.64) | 8 (72.73) | 8 (72.73) | 8 (72.73) |
| Tier 2 justification in line with RI (%) | 6 (54.55) | 8 (72.73) | 6 (54.55) | 6 (54.55) | 6 (54.55) | 8 (72.73) |
Only in two items 72.73% of the participants choose their answer in line with RI in the second tier, while in the other four items only 54.55% of the participants choose their answer in line with RI in the second tier. For this reason we readjusted the item scenario as well as tier one.
After revising the pilot test, a second pilot run was conducted with four master students. These students were given the P2I questionnaire (P2I Questionnaire S, Vers. 1.0) (Zollitsch et al., 2020b) in a pre-post-test design. They all came from interdisciplinary studies in Coburg University in Germany, and the age range was between 23 and 32 years. The group was not randomised and the participation was voluntary. To have a full pre-post-pilot run, subjects who participated in the first interventions of the P2I training were asked to conduct the questionnaire before and after their training. They were personally contacted by the instructor of the P2I training. Since the test was voluntary, only four participants finished the pre- and the post-test. Nevertheless, this test run exemplified that the complete questionnaire design, as presented in the P2I Questionnaire S, Vers. 1.0 (Zollitsch et al., 2020b) works.
Figure 2 shows that in the adjusted questionnaire, more participants chose a practice in line with RI in the post test in the items cp5 and cc5. In the items cc1 and cd2, the participants answered with the same frequency choosing practices in line with RI in the pre-test as well as in the post-test, while for the items cp3 and cd9 practices in line with RI have been chosen less often in the post test.
Figure 2. Comparison of the results of a pre- post-test.
Acknowledging that the test did not assess learning successes as wished, we decided to rearrange the structure by using the above-described tier models and rewrote the failing items. We modified the original structure. Instead a binary response in the first tier (decision between yes or no), the new version of the P2I questionnaire includes four possible options. At the same time, the justification patterns (not) in line with RI that were previously based on whether the practice corresponds to a practice in line with RI or not were summarised from four justifications per answer (meaning eight in total: four yes/four no) into a total of four justification patterns. Thus, someone who chooses a practice not in line with RI can still select a rationale that fits a justification pattern in line with RI.
The first tier no longer requires testees to decide whether a described practice is or is not in line with RI, but rather testees must choose which of the four given practices is in line with RI.
We expanded the P2I questionnaire’s multiple-tier approach to evaluate in detail if the testees can determine which practice is in line with RI and how to justify this decision. We added two confidence tiers (Caleon & Subramaniam, 2010; Peşman & Eryılmaz, 2010) to create a so-called four-tier structure. This approach is used to find out if testees are guessing their answers. These confidence tiers determine how sure the testees are with their responses. Information from four-tier tests provides valuable insights into the development and improvement of training. They enable conclusions, especially in cases of answers not in line with RI, for example, if testees are using a specific justification more often than others. Data gives insights into if there is simply a lack of knowledge – this would be expressed in low confidence – or whether the problem lies in a stable pattern (not) in line with RI. (P2I Questionnaire S, Vers. 2.0 (Zollitsch et al., 2020b) and P2I Questionnaire M, Vers. 2.0 (Zollitsch et al., 2020a))
Test for pre-validating and revision
To pre-validate the final four-tier structure of the P2I questionnaire, the developers conducted a content validation test (Miyata & Kai, 2009) with international research integrity and research ethics experts. In the first step, the RI/RE experts gave feedback on the content. In a next step, the validation test followed the hypothesis that international experts would respond with high confidence to the P2I questionnaire; and students (without specific training) would respond with less confidence than experts.
The test developers invited all 17 members of the Horizon 2020 project Path2Integrity and 12 additional research integrity and research ethics experts working with the training centre of Path2Integrity to validate the content. Invitations were extended via email only to those who did not have a connection to the learning units or the P2I questionnaire. Their credentials were accredited via their respective position in either the P2I advisory board, the P2I scientific board, or by being a contact from the P2I training centre. The validation procedure instructed all experts to complete both versions (the S and the M/Y version), to give feedback, to make the questionnaire’s content valid as possible, and ask questions when needed. The participation was voluntary, and the group was not randomised.
We received 27 responses to the P2I questionnaire from the experts' group mentioned above. The experts came from various disciplines (including Humanities, Medicine, Law, and Natural Sciences) around the world. The experts’ group was gender-balanced, and the age structure varied from 35 to 67. Each expert received both versions of the questionnaire (P2I Questionnaire S, Vers. 2.0 ((Zollitsch et al., 2020b) and P2I Questionnaire M, Vers. 2.0 (Zollitsch et al., 2020a)) and all of them were asked to answer both and comment on them. 12 experts filled out version S (P2I Questionnaire S, Vers. 2.0) (Zollitsch et al., 2020b), and 15 filled out version M / Y (P2I Questionnaire M, Vers. 2.0) (Zollitsch et al., 2020a). From those 27 replies, only 12 contained content-specific comments. The test developers checked these comments and integrated them into the P2I questionnaire.
The experts’ comments did not only refer to the content but rather often to aspects of design. For example, having an open field for the category country, not storing IP addresses, and providing a more prominent link to the data policies. We implemented those recommendations accordingly. One comment stated that items for data management are missing in version S. As shown in ‘table 1. Content if the P2I questionnaire (referring to the European Code of Conduct)’ data management is a topic for those who graduated. Therefore, we did not integrate those items in version S. Furthermore, two comments declared that they are missing the handling of dilemma situations. As explained in the status quo, this questionnaire does not focus on moral dilemmas but rather on contradicting practices, and on arguing in favour of research integrity, and therefore, does not contain dilemma situations. On top of that, experts commented on the test's difficulty; they stated that some items were difficult to answer.
Two comments directly addressed the content of one item, namely S1. These comments led to an item amendment. The developers extended the option "ombudsperson" to "ombudsperson/research integrity officer" because these terms are synonyms but used exclusively in some countries. Another comment was on item S4. Experts asked the developers to clarify the case as well as the answers. The developers shortened the item from "Sarah is in charge of the coordination of a research project" to "Sarah is in charge of a research project". Also, the answer "informs each group that it is responsible for establishing and adhering to its own principles" was changed into "informs each partner that they are responsible for establishing and adhering to their own principles". A third expert comment asked to give more than one response option. The developers declined this comment after discussing and weighing it against the central concept of the P2I questionnaire, since the participants should decide on only one practice and justify this practice. If they are not sure about their answer given, they have always the option to tell by using the slider in the second and fourth tier.
The procedure confirmed the content validity of the P2I questionnaires because none of the experts questioned the validity of the test and the few content-specific comments were adjusted in consultation with the experts. As there were no indications that other RI and RE experts would come to a fundamentally different conclusion regarding the content validity of the questionnaire, we considered the number of participants to be saturated and refrained from consulting further experts.
In the next step, we contacted six lecturers from Kiel University in Germany to acquire students from non-RI/RE related courses. Two lecturers offered 15 students that had not already completed the test. The P2I questionnaire was administered by the lecturers personally. The participation was voluntary, and the group was not randomised. Ten students participated in this second step of the validation study (five in Version S (P2I Questionnaire S, Vers. 2.0, (Zollitsch et al., 2020b)) and five in Version M (P2I Questionnaire M, Vers. 2.0)) (Zollitsch et al., 2020a). The students' group consisted of five bachelor students and five master students. They were between 23 and 29 years old, with seven females and three males (from Germany); most of them from the field of social science, some from humanities and economics.
The following describes the confidence (tier two and four) for a small sample of items from the S version of the P2I questionnaire as an example. Beyond these examples described, we pre-validated all items (and both versions) of the P2I questionnaire. Figure 3 shows the students' confidence with their answers from the first tier and the third tier combined compared to the experts' confidence (for the items in version S). (P2I Questionnaire S, Vers. 2.0) (Zollitsch et al., 2020b)
Figure 3. Comparison of the confidence of experts and students.
Figure 3 shows the contrast of the average student's confidence and the average expert's confidence and the median. Students are less confident with their decisions in the first and in the third tier, except for Item S3. The median in all items is higher for the experts than for the students. Also, the range of confidence is much broader in the students' group than in the experts' group, which displays that the students might be confident within some decisions but not in all research integrity topics.
The result of this comparison is not ideal, because experts’ confidence does not concede students’ confidence in one out of five items. Nevertheless, comparing how confident the students and experts responded showed that students, in general, are less confident than experts. This comparison documents that the P2I questionnaire is (only) a pre-valid instrument to assess research integrity training, as the students are not as familiar with research integrity as experts.
Results
The result of the development and validation is a pre-validated P2I questionnaire, which needs to be validated in a second pilot study. The following describes the P2I questionnaire in its latest content and structure (P2I Questionnaire S, Vers. 2.1 (Zollitsch et al., 2020b) and P2I Questionnaire M, Vers. 2.1 (Zollitsch et al., 2020a)).
As shown in Figure 4, the P2I questionnaire opens with a scenario followed by a choice of which practice is in line with RI. This decision in the first tier consists of a multiple-choice question with four possible options to choose from. The first tier asks what the person in the scenario should do to follow a good research practice. The questionnaire’s second tier asks on a scale from 1 (very unsure) to 100 (very confident), how confident the testees were with their choice in the first tier. In the third tier, testees are asked a multiple-choice question with four alternatives to justify the practice chosen in the first tier. Again, the next tier asks on a scale from 1 to 100 how confident the testees are about their third-tier justification.
Figure 4. Exemplary item of the P2I questionnaire from the field of collaborative working.
The confidence tiers are essential for determining if the testees are confident about both responses in tier one and tier three. Ideally, they choose a practice in line with RI, justify this choice with a justification in line with RI while confirming that they are confident with both answers.
To consider relevant differences that might occur, the P2I questionnaire also contains different independent variables to evaluate RCR training successes next to the tiers’ responses.
Discussion
As stated earlier, one of RCR training’s core themes is to enable participants to become a part of the research community. One way to assess this is to observe what practices (future) researchers choose in their research environment and how they justify their choice. As shown in the three different steps above, the assessment of justifications is – in particular – difficult. The P2I questionnaire is a first attempt to assess such practices and justifications (not) in line with RI.
The P2I questionnaire derives from Path2Integrity’s RCR training, compared to the DIT-2, which originates from ethics education. The P2I questionnaire focuses on practices in line with RI in reference to the research community – in particular – in reference to the ECoC (2017). Both instruments start with a (case) scenario that testees should respond to. While the DIT-2 asks which practice is the most important, the P2I questionnaire asks to justify this practice. The P2I questionnaire, therefore, gives insights how testees argue in favour of specific practices.
The multiple-tier structure is a frequently used design for assessing RCR related topics in the Asia-Pacific region. In other educational fields, it is often used for natural science education, such as Geology (Chang et al., 2020), Physics (Winarti et al., 2017), or Mathematics (Fratiwi et al., 2017). Based on the ideas of Chou et al. (2007) or Sun (2009), the P2I questionnaire transferred this multi-layered design to assess RCR training’s successes.
Although the P2I questionnaire has been developed in the Horizon 2020 project Path2Integrity, it will also be applicable to other RCR training. The P2I questionnaire’s framework ECoC (2017) is expandable and applicable to other reference documents such as the Singapore Statement, 2010, the Montreal Statement, 2013 etc. These codes refer to the same practices in line with RI and entail similar justifications. The P2I questionnaire focuses on its assessment on practices and justifications in favour of research integrity. It is therefore suitable for RCR training that focuses on dialogical approaches (Koterwas et al., 2021).
The P2I questionnaire is a first step to offering a standardised instrument with which the RI community can achieve reproducible results and compare different RCR training. This article contributes to the dialogue about standardising and standardised instruments ton assess RCR training.
Limitations
The P2I questionnaire is a specific tool with limitations. It demands a high reading competence of the testees, as it is very much text based. Users need to consider that they also measure the literacy of the testees. Due to the complexity of the four-tier test structure, some testees also may drop out during the evaluation. Limitation of the pre-validation and testing: No secondary school students designed and pre-validated the P2I questionnaire. The leading group of involved participants – next to the 27 responses of international experts - affiliates to educational studies, and most of them identified their country as Germany.
This article described the development of the P2I questionnaire following Chandrasegaran, Treagust, and Mocerino’s (2007) approach, designing and pre-validating the P2I questionnaire in three steps. As shown above, the expert student comparison did not receive ideal results. That is why further research with the altered P2I questionnaire version is needed. Before the questionnaire is used as a standardised assessment instrument for learning success in RCR training, a second validation is necessary.
Data availability
Underlying data
The data of all parts of the pilot test cannot be shared because it might jeopardise the achievement of the main objective of the Path2Integrity project as described in H2020 MGA article 29.3. As the data contains answers especially from experts, these might be used as orientation from future participants of the P2I test and this will jeopardise the result. All data will be shared at the end of the project, latest at the end of June 2022 in the data project included in the data statement below, Zenodo.
Zenodo: P2I Questionnaire Version MY, https://doi.org/10.5281/zenodo.4682660. (Zollitsch et al., 2020a)
This project contains the following underlying data:
P2I questionnaire MY Vers_1.pdf
P2I questionnaire MY Vers_2.pdf
P2I questionnaire MY Vers_2.1.pdf
Zenodo: P2I Questionnaire Version S, 10.5281/zenodo.4680216. (Zollitsch et al., 2020b)
This project contains the following underlying data:
P2I questionnaire S Vers_1.pdf
P2I questionnaire S Vers_2.pdf
P2I questionnaire S Vers_2.1.pdf
Extended data
Zenodo: Draft of the P2I questionnaire, https://doi.org/10.5281/zenodo.5059949. (Zollitsch et al., 2019a)
Zenodo: Pre-Version of the P2I questionnaire, 10.5281/zenodo.5059965. (Zollitsch et al., 2019b).
The data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).
The data of all parts of the pilot test cannot be shared because it might jeopardise the achievement of the main objective of the Path2Integrity project as described in H2020 MGA article 29.3. As the data contains answers especially from experts, these might be used as orientation from future participants of the P2I test and this will jeopardise the result. All data will be shared at the end of the project, latest at the end of June 2022 at Zenodo. If prior access to the data is desired, this can be granted on the condition that neither the data nor analyses based on the data are published in any way. If access to the data is required before 30.6.2022, please contact the first author of the paper Linda Zollitsch (zollitsch@path2integrity.uni-kiel.de).
