Introduction
Citizen science, defined as collaborative research with a varying degree of involvement of citizens in scientific processes (c.f. Heigl et al., 2019), is not a recent phenomenon. Even if it was not known by the name ‘citizen science’ in the 19th century, aspects of the approach can be found in earlier forms of collaboration between scientists and lay people. For instance, the Christmas Bird Count, initiated by the National Audubon Society in 1900, is recognised as one of the oldest and most notable citizen science projects (Dunn et al., 2005). It involved volunteers documenting bird species and populations during the winter season. With growing environmental awareness, citizen science projects focusing on the monitoring of pollution and ecological changes began to emerge. Notable examples include the Cornell Lab of Ornithology's Breeding Bird Survey (started in 1966) and the Community Collaborative Rain, Hail, and Snow Network (CoCoRaHS) established in 1998. The advent of the internet and digital technologies subsequently revolutionised citizen science. Online platforms, such as Zooniverse, launched in 2007, allowed volunteers to contribute to various research projects through the analysis of large datasets and images.
What is new in the rise of the modern form of citizen science is a more radical involvement of volunteers in the scientific process, questioning the traditional relationship between scientific knowledge production and its reception (e.g. Delfanti, 2010). The idea of citizen science holds the idealistic promise to bridge the gap between scientists and citizens, with benefits on both sides (Robinson et al., 2018). While the role of lay people was merely limited to assisting in the collection of data in early collaborations, the degree of involvement of volunteers in current citizen science projects varies, with their inclusion in different phases throughout the research process.
Citizen science is about democratisation of access to science (Curtis, 2018; Giardullo et al., 2023; Peters & Besley, 2019), breaking up the so-called ‘ivory tower’ of science, and an empowerment of citizens in the scientific undertaking (Herzog & Lepenies, 2022). Researchers implementing citizen science projects acknowledge how involving citizens brings in different perspectives and that some of the responsibilities and duties in the research process are being shared (Shirk & Bonney, 2018).
The participation of citizen scientists ranges from active engagement in scientific activities and processes, to contributions to evidence-based policy evaluation and development (Haklay, 2015; Rowland, 2012).
Levels of involvement and engagement vary, depending upon the type of citizen science project and the stage of the research process (c.f. Figure 1).
Figure 1. Stages of the scientific process non-scientist participants are involved in according to the type of citizen science project, adapted from West and Pateman (2017, cited after Reynolds et al., 2021).
According to the so-called ‘extreme citizen science’ approach (Chiaravalloti et al., 2022), citizens should be involved in all phases of the scientific process, from co-defining research questions to disseminating results. People with different educational backgrounds should be able to access and participate in citizen science projects so that a true democratisation of science can take place. ‘Extreme’ here refers to the extent of participation and to people who were previously excluded from the production of scientific knowledge. The ‘extreme citizen science’ approach calls for inclusive projects that are accessible regardless of background, counteracting the pattern of ‘white, well-educated and male participants’ in citizen science projects (Curtis, 2018).
Thus, citizen science projects require open scientific practices and a positive attitude towards citizen science by the researchers. However, one can pose the question, why should citizens join and what’s in it for them? And are potential benefits true for all?
One of the most prominent motivational drivers for citizen scientists, according to a number of reviewed citizen science projects, is the desire to contribute to a “greater good” and to help science to solve problems that are perceived as relevant and meaningful in today’s society (Land-Zandstra et al., 2016; Martin et al., 2016; Silva et al., 2016). Once citizen scientists are active and involved, intrinsic motivators and social influences gain more importance in keeping them active and engaged (Nov et al., 2011) and in increasing not only the quantity, but also the quality of their contributions (Nov et al., 2014).
Assessing benefits for citizen scientists
Although appreciation for citizen science projects continues to grow, the number of projects demonstrating the impact on involved citizen scientists and diverse demographic characteristics is still limited. Outcomes of the collaboration between citizen science and research can be manifold, depending on the type of project (e.g. Bonney et al., 2009) and the involvement of citizens at different stages of the project. As such, the evaluation of project goals and outcomes can also be manifold. Up to now, by far the most commonly investigated scientific outcomes of citizen science initiatives concentrate upon the number of related scientific publications. Another aspect that can be observed in various types of citizen science projects is the development of new skills and knowledge by citizen scientists (e.g. Schäfer et al., 2020; Wiggins & Crowston, 2015). Firstly, there are references to the importance of knowledge gains related to the research topic as being the most important impact for participants (Stepenuck & Green, 2015). Secondly, the involvement in citizen science activities teaches the participants about the process of scientific enquiry and helps them to gain a deeper understanding of scientific outcomes (Bela et al., 2016; Richter et al., 2016; Riesch & Potter, 2014). The citizen science approach inspires stewardship, and enhances the sense of participant empowerment (Crall, 2010; Crall, 2013; Sutcliffe, 2011; Wickson & Carew, 2014).
Experts recommend defining specific goals, expected learning outcomes, and a customised evaluation strategy with measurable indicators (Jordan et al., 2012; Phillips et al., 2014). For the evaluation strategy, the pre-existing knowledge and skills of the target groups must be aligned with the expected learning outcomes, in order to be able to properly assess participant learning gains and assess the impact of the project (Skrip, 2015).
Citizen scientists have been directly approached to assess what they felt to be the benefits of their participation in projects and to measure their learning outcomes. Cox et al. (2015) differentiate between contribution to science and public engagement. Others have released guidelines on how to set-up a citizen-science project, including recommendations regarding their evaluation, such as Bonney et al. (2009), who suggest the evaluation of scientific literacy outcomes through the use of similar indicators, such as the duration of involvement by project participants; the numbers of participant visits to the project website; but also direct surveys directed at citizens, in order to measure how understanding of science content and of science processes improves, etc. A study on measuring outcomes in citizen science projects, Bonney et al. (2016), found, through surveys of citizen-science practitioners and additional interviews, the following constructs to be achievable and measurable: interest in science and nature; self-efficacy for science and environmental action; motivation for science and environmental action; science enquiry skills; data interpretation skills; knowledge of the nature of science; and environmental stewardship. To support the evaluation of citizen science projects, the Cornell Lab of Ornithology elaborated evaluation guidelines that focus especially on learning outcomes, such as the acquisition of new knowledge and skills, but also on increased interest in science, motivation, self-efficacy in science-participation, personal development and behavioural change (Phillips et al., 2014).
The most frequently used evaluation instruments are not only survey interviews, and the analysis of participant communication (Gommerman & Monroe, 2012), but also stakeholder consultations, observations, iterative adaptations with actors in the field and self-assessment tools applied during the evaluation process (Kieslinger et al., 2017). As Peter et al. (2019) stated in their report about citizen science projects, many studies rely on self-reported data (which is not very precise, due to social desirability bias) in their evaluation designs and strategies.
This overview illustrates that there is no single proven approach for the evaluation of citizen science projects, since all approaches comprise useful elements.
In this study we investigate, based on four citizen science projects in the fields of astrophysics and particle physics, the democratisation of access to these projects as well as their gains and how these are related to demographic variables, concentrating on outcomes on individual level.
The specific research questions for the paper are:
(i) What do citizen scientists gain in terms of attitude towards science, motivation to join, self-efficacy, scientific skills, and knowledge acquisition in astrophysics and particle physics, in the four citizen science projects?
(ii) What are the citizen demographics and characteristics, such as gender, age, educational background, experience of discrimination as an (indicator for democratised access and inclusion)? Are there differences between different groups in terms of their gains (indicator for equal benefits)?
In the following sections we briefly introduce the four citizen science projects implemented on the Zooniverse platform. We then describe the methods used to collect evidence of how participants benefited on a personal level, before providing the overall results. In the discussion and conclusion section, we aim to answer the two research questions and hint at further research gaps.
Four citizen science projects on Zooniverse
The four citizen science projects were developed as part of the REINFORCE project, which was funded within the Science With And For Society theme of the EU Horizon 2020 framework and were implemented on the Zooniverse platform (https://www.zooniverse.org/). On Zooniverse, volunteers interested in participating in research can contribute online to different projects across a broad range of research areas. The platform also encourages citizens to enter into dialogue with research teams on the respective discussion platforms, known as ‘Talk-pages’ to clarify open questions and concerns.
The four citizen science projects can be classified as a mix of contributory and collaborative, rather than co-creative citizen science projects (c.f. Figure 1) as citizen scientists do not collect data but analyse already collected data that is shared by large research infrastructure. According to Shirk and Bonney (2018) the projects on the Zooniverse platform are a primary example for the “technology transfer” category, in which digital images can be classified by citizens and can contribute to citizen science projects. As the exercises to be done mainly involved categorisation tasks, opportunities were provided to interact with researchers and other citizen scientists and to ‘dive deeper’ in the respective fields of research. These comprised online interaction options (online forum, webinars, online visits to research infrastructure) as well as face-to-face events (public lectures, course for seniors, artistic interventions).
The REINFORCE implementation period of the four citizen science projects ran from the 19th of October 2021 to the 25th of October 2022, although all four of them are still available online.
GWitchHunters
The GWitchHunters1 project developed an advanced citizen science programme by providing access to representations of gravitational wave (GW) data produced by the Virgo2 detector. These included data taken from the GW strain channel, as well as from auxiliary channels, providing environmental background information. Since the sensitivity of GW detectors is limited by several types of noises, it is crucial to understand their origin and impact on data acquired. By ‘hunting’ for noises and systematically profiling them, the research team can undertake in-depth analyses and contribute to the development of a more efficient detector with a wider detection span. The citizen scientists in GWitchHunters contributed to this activity by looking at chunks of data and identifying transient noise artefacts, known as ‘glitches.’ The outcome of these activities was used to train machine learning algorithms to automatically recognise and isolate these glitches in GW detector data.
The GWitchHunters (see Figure 2) research team also collaborated with a sister project, GravitySpy3; also available on Zooniverse, which is a highly successful citizen science project developed using data from the LIGO4 detectors, based in the United States.
Figure 2. Screenshot of GWitchHunters on Zooniverse.
Deep Sea Explorers
In the Deep Sea Explorers5 project on Zooniverse (see Figure 3), citizen scientists helped to optimise the categorisation of data gathered by the KM3NeT neutrino telescope, which collects environmental noise from the bottom of the deep sea in two locations: one to the south of France; the other off the coast of Sicily, Italy, in the Mediterranean Sea. By participating in the analysis on Zooniverse, citizen scientists were able to engage with the world of neutrino astronomy and gain at the same time insights into the unexplored deep marine environment. In the framework of the Deep Sea Explorers project, citizen scientists were invited to classify different sources of bioluminescence, recorded by the detector, and bioacoustic signals registered by hydrophones situated in the surrounding environment.
Figure 3. Screenshot of Deep Sea Explorers on Zooniverse.
New Particle Search at CERN
The New Particle Search project at CERN6 engaged citizen scientists with data recorded by the ATLAS detector of the Large Hadron Collider (LHC) at CERN (see Figure 4). For the purpose of the New Particle Search project, the researchers developed a specific software for the display and analysis of the ATLAS data (HYPATIA7) and asked the citizen scientists to look for evidence of undiscovered particles. On the platform, the citizen scientists were able to classify static images, interact with the event display, select specific tracks, and calculate invariant masses. Some particle decays, such as photon conversion, were more accurately identified by humans than by algorithms and, with the aggregated data from thousands of citizen scientists, the researchers had the possibility to explore and examine their data further. The data categorised by citizen scientists can be compared to the categorisations produced by machine learning algorithms and can serve as a baseline for further research. The citizen scientists received valuable feedback from the New Particle Search researchers and were able to draw their attention to interesting events for further investigation.
Figure 4. Screenshot of New Particle Search at CERN on Zooniverse.
Cosmic Muon Images
The Cosmic Muon Images8 project focuses on interdisciplinary studies involving geoscience and archaeology and aimed to show how technology can be used to study fundamental physics and develop frameworks that have a significant impact on society (see Figure 5). In the project, researchers provided citizen scientists with an open data set, recorded by cosmic-ray detectors during a period of data-taking at the Apollonia Tumulus in Greece, in 2018, and invited them to interact with the data and make classifications.
Figure 5. Screenshot of Cosmic Muon Images on Zooniverse.
Inclusion strategy
In order to reach diverse audiences and to engage them in the projects as citizen scientists, the engagement strategy of REINFORCE comprised specific approaches for elderly people and people with visual impairments. These target groups were specifically addressed to include participants who are often left out in citizen science initiatives.
Inclusion strategy for elderly citizen scientists via specifically designed courses
As part of the REINFORCE project, a course on science specifically for senior citizens, was organised in collaboration with the organisation Università della Libera Età (University of the Free Age), based near to the Virgo gravitational-wave detector, in Cascina, near Pisa. The course was structured around approximately monthly sessions, given in-person, at the home of the University group: the municipal library of Cascina. The first implementation closed with a visit by more than 40 members of the group to the European Gravitational Observatory (EGO), the home of the Virgo detector, and led to a second edition of the course being implemented over the following academic year, well beyond the natural lifetime of the REINFORCE project itself.
Course sessions were delivered by professors and researchers from the University of Pisa and members of the REINFORCE collaboration based at EGO. Most of the sessions were delivered in Italian, as most of the group were mother tongue Italian speakers and did not speak a second language. Despite this, two of the sessions - those dedicated to the cosmology of the (in)visible Universe and that on art and science - were delivered in English, were both very well received by participants, leading to the conclusion that, where sessions material was potentially more accessible, especially when the themes covered were supported with visually explicative presentation, it was better suited to cut through and hold participant attention even when delivered in a different language.
Inclusion strategy for citizen scientists with visual impairments through sonification of data
Increasing the senses, increasing inclusion, was the “big argument” for the development of a software tool - sonoUno - dedicated to data sonification. The ambition of this work is to expand the senses used in scientific inference, beyond the visual, and to include in the general effort of the scientific community, people with sensory disabilities (especially visual).
In sciences such as astrophysics and physics, scientists constantly interact with numerical data, generally represented visually. These interactions imply a response that is mostly related to current events, and which are limited by the data analysis tools available and the resolution of display elements such as screens. There are studies (Diaz Merced, 2013) that show that multisensory display of data can improve signal detection, especially if it is astronomical data. This allows us to infer that a sound recording alongside its visualisation can contribute to a better understanding of results, and, as such, can allow people with functional diversity to analyse scientific data, and then contribute to scientific discoveries.
A first training workshop in data detection by sound (performed in August 2022) allowed us to obtain important results, as well as to confirm earlier ones, and evaluate the performance of a group with the new tool. It has become evident that the possibility to use sound improves the integration of people with disabilities in the study of science and the multimodal approach helps the understanding of conceptual and scientific content, allowing the same phenomenon to be explored through different sensory channels, for disabled and non-disabled people alike.
These two strategies were developed in addition to the general engagement strategy of the project with dedicated seminars, virtual tours, and outreach via social media channels and the projects websites.
Methods and materials
In our study we have, together with the research teams, discussed potential gains in the four astrophysics and particle physic domains, in the sense of Skrip (2015) who, as described above, suggests that learning outcomes must be defined beforehand. To structure the different evaluative dimensions, we relied on the logic model by Kurz and Kubek (2016), which although not specifically designed for citizen science projects, is a useful instrument to differentiate between outputs, outcomes, and impacts (cf. Table 1).
Table 1. Logic model of Reinforce citizen science projects.
| 1 Outputs (what we do) | 2 Outcomes (results at target group level) | 3 Impact (results at
societal level) |
|---|
1a Output
● Web-based interface
(Zooniverse)
● Sonification
● Citizen education (citizen training
activities) e.g. vision building
workshop, online and in-situ
training, practice reflection, etc.
● Community empowerment
and awareness activities e.g.
workshops, summer/ winter school, Science café, open
schooling day, etc.
● Educational resources | 2a New knowledge, skills,
attitudes and awareness
Citizens Scientist:
● new knowledge: comprehend role of large RI;
understanding basic physics concepts; principles of
machine learning; methods of scientific investigations
● skills: data recognition and analysis skills, critical thinking
● attitude: awareness of science/scientific work (e.g.
collaboration, daily scientific processes); inclusiveness of
science; increasing awareness of the interaction with nature;
identification with their direct contribution to science
Researcher:
new knowledge: further research insights; gaining more
experiences with citizen science projects and their potential for
future CS projects | 3 Social and economic
impact
● Enhance science literacy
of the society, public
understanding of
science, critical thinking
● Economic costs and
benefits of citizen
science
● Enablers and barriers
for development of
new knowledge citizen’s
science
Science career motivation |
1b Use of output by target groups
● Participation in Zooniverse
projects
● Participation in Citizen education
● Participation in Community
empowerment activities
Reach of different target groups
(elderly, visual impairments, pupils) | 2b Change actions/behaviour
● Science career motivation
● Cooperation with researchers
○ Collaboration between citizen scientists and researchers
○ Experience exchange among citizen scientists
Improving mutual understanding through exchanging diverse
expertise on a larger scale |
1c Participants satisfaction
● Zooniverse project experience
● F2F event (Citizen education;
community empowerment) | 2c Living conditions
● Citizens feel empowered by contributing to science
● Participation is possible even in confinement (e.g. Covid-19) |
According to the logic model outputs are what the projects offer, its use and the participants’ satisfaction; outcomes are what the project aims to achieve with a target group; and impacts are the contributions of the project on a societal level. Applied to the four citizen science projects in discussion with the four research teams is a general logic model valid for all four citizen science projects with an additional layer of customisation, according to the specific fields of research, which built the basis for the later operationalisation of the different dimensions.
To measure the outcomes on the individual participant level, a one-group pre-test/post-test design (Levine & Parkinson, 2014) was implemented. This design falls under quasi-experimental designs as the main premise of true experiments, namely the existence of a control or comparison group and the random selection and assignment of participants, is missing. As a result, although one would be able to assume that changes from the pre-test to the post-test are due to the participation of citizen scientists in the citizen science projects, unlike in true experiments, in which such effects would be solely attributed to this participation; in this design, outside factors cannot be controlled or ruled out. Nevertheless, this design is more reliable and provides more accurate data than a one-group post-only design, which, due to the lack of a pre-test, cannot show any change in relation to skills, knowledge, attitudes, behaviours, level of awareness, etc.
Constructing the pre/post questionnaire
The development of the questionnaire involved the following steps (see overview in Figure 6): (1) desk research on evaluation surveys in citizen science projects with a similar focus; (2) a compilation of items from different already available surveys that were suitable for our purposes; (3) first selection of items; (4) alignment with the general logic model and the specific logic models of each of the four citizen science projects, respectively; (5) compilation of a draft version; (6) collection of feedback by research teams and user testing with 10 volunteers; (7) integration of feedback; (8) update of another draft version; (9) implementation on the online survey tool LimeSurvey9; (10) accessibility check by people with visual impairments; (11) cognitive pre-testing (Prüfer & Rexroth, 2005) with potential users and ‘thinking aloud’ protocols to test the usability aspects of the items and detect potential misunderstandings; (12) last fine-tuning of the questionnaire and integrating the link to the survey in the four Zooniverse projects.
Figure 6. Development of the pre- and post-questionnaire.
Desk research on citizen science studies (c.f. step 1) focused on the evaluation of citizen science projects and Zooniverse projects in particular, and their evaluation instruments and questionnaires, respectively. As far as our desk research indicates, only rarely has a pre-post design been used in Zooniverse projects. In their survey on Planet Hunters (Depper, 2019), another Zooniverse project, citizen scientists' learning gains were assessed retrospectively, through a question on whether volunteers felt they had learned anything and, if so, what they had learned. This post-only design obviously has several flaws; not least individual memory capacity, which might influence the results (Medici et al., 2010).
The User’s Guide for Measuring Learning Outcomes in Citizen Science (Phillips et al., 2014) released by the Cornell lab for Ornithology, has become one of the standard resources in the quest to assess citizen science project outcomes on a participant level, including several questionnaires that have already been widely used and which have been checked in terms of their quality, such as reliability and validity (Phillips et al., 2018). Some of these questionnaires have been framed as general questionnaires, which can be used mostly for any kind of citizen science project, and custom questionnaires, which can be adapted to the specific context. Thus, the questionnaires constitute a valid source for our study. All questionnaires were screened and those items that were in line with the logic model (c.f. step 3 and 4 in the development of our questionnaire) were extracted. Other additional questions resulted from “inspiration” from the other evaluation studies cited above and from the interaction with, and feedback provided by, project partners and demonstrator research teams and the citizen science expert team (c.f. steps 5, 7 and 9).
In the pre/post-test design as applied in our study, the pre-survey aims to measure the “baseline.” In our case, this covered five distinct areas: knowledge (self-reported and tested knowledge), motivation to join, self-efficacy, skills, and attitudes. Each of these dimensions comprises five and 19 items (cf. Extended data, complete questionnaire; Unterfrauner & Fabian, 2024).
Thus, eventual changes in these areas could be assessed in comparison with the post-survey. Although the four citizen science projects were slightly different in nature, they each required the performance of broadly similar tasks, i.e. the classification of data representations. The field of research differed, however, and this was reflected in the construction of the questionnaire knowledge items, where the single items differed slightly across the four citizen science projects.
The questionnaires were implemented in LimeSurvey, and the collected data imported for further analysis in the statistical analysis software SPSS.
The link to the pre-questionnaire was included on the project home pages and in the Zooniverse project tutorials. Participants read through the tutorials in order to understand how to classify data and contribute to the individual projects. Participants were asked to share their email addresses when they first filled in the questionnaire and received an email with a link to the post-questionnaire one month after filling in the pre-questionnaire. For technical reasons it was not possible to include the post-questionnaire on Zooniverse with impact on response numbers (see below).
The data collection using the surveys for each of the projects started when they were each launched as official projects on Zooniverse. New Particle Search at CERN became an official Zooniverse project on the 26th of October 2021; GWitchHunters on the 16th of November, 2021; Cosmic Muon Images on the 11th of January, 2022; and Deep Sea Explorers on the 8th of February, 2022. The data collection period ended on the 17th of August 2022.
Results
Analyses of participant data showed that there were no significant differences in overall proportion between responses from the four projects. Consequently, the four sets of responses were merged into an overall dataset for further analysis (Unterfrauner, 2024). The limitations of the data lie in the fact that, as with all voluntary participation, the dataset is limited to only those people who took the time to complete the questionnaire and thus represents a self-selected sample (Bethlehem, 2010). The data might therefore not be fully representative of all participants in the four projects. Nevertheless, they shed light on the demographics, gains in knowledge, skills, and changes in attitude towards science and allow for a differentiated analysis of changes, taking into account demographic aspects and experiences of discrimination.
The analysis of the demographic characteristics of the citizens participating in the four projects is based on the data of the pre-questionnaire, in order to give a more comprehensive picture, while, for the comparison of pre-questionnaire and post-questionnaire scores, it was possible to use only complete datasets, i.e. where the respondent had completed both questionnaires.
While the pre-questionnaire was filled in by a total of 1,179 participants, the post questionnaire had only 301 responses, resulting in a response rate of 25.5% in the second round. In the following only complete datasets have been taken into account.
Demographic characteristics
The analysis of the gender composition indicates that 53% were male, 40% female, 3% preferred not to say and 4% defined themselves as non-binary. In terms of age (c.f. Figure 7) and educational level (cf. Figure 8), participants were quite diverse. About 13% were below the age of 20 and a small fraction, i.e. 1%, were above the age of 80, while the remaining age classes, between 20 and 80 years old, were fairly equally represented in the sample.
Figure 7. Demographic data - age.
Figure 8. Demographic data - education.
A closer look at the composition regarding educational level reveals that the sample is skewed towards higher education degrees, including Bachelor’s, Master’s and PhDs or other advanced professional degrees. However, also people with lower educational degrees accessed the projects and were able to contribute. More than one third of the participants had not completed an academic degree.
More than half indicated that they had no professional background in science (52%), 38% had a scientific background and 8% were not totally sure.
A few participants (5%) indicated a visual impairment, which required assistive technology and could not be compensated for with glasses, and about a quarter of all participants indicated that they felt they belonged to a discriminated against group. The discrimination experience is attributed to multiple different factors (c.f. Figure 9), the majority feeling discriminated against for their gender identity and sexual orientation, followed by disability, ethnic group and migration history.
Figure 9. Demographic data – discrimination experiences.
Engagement level
On average, the respondents made 109 classifications (with a standard deviation of 826 and a maximum of 24,179 (!!) classifications) and spent 2.09 hours on the project (standard deviation of 14.8 hours and a maximum of 160 hours)10. The average classification time was 60 seconds (with a standard deviation of 112 sec and a max of 595 sec).
The level of engagement in terms of performed classifications and time spent on the Zooniverse projects varied to a great extent. Thus, engagement levels are also taken into account in the analysis of changes.
Changes in motivation, attitudes, knowledge, self-efficacy, and skills
In the following, we describe the changes resulting from the comparison between pre and post-test scores on the dimensions: attitude, motivation to join, scientific skills, self-efficacy in relation to scientific undertaking, reported scientific knowledge and tested scientific knowledge.
The pre-questionnaire served to measure the ‘baseline,’ i.e. scores before participating in the citizen science projects, to compare with the scores from post-questionnaires, which were filled in after one month of participating in the citizen science projects.
The following table shows the results of the paired sample T-tests. The first five subscales and resulting scores are based on a 5-point Likert scale, from 1=strongly disagree to 5=strongly agree. In the last subscale on tested knowledge there were three answer options, i.e. ‘yes,’ ‘no,’ and ‘don’t know.’ The score on tested knowledge can range between 0 and 1, indicating the ratio of correct answers.
The following overview in Figure 10 shows the average scores (means) per evaluation dimension, both in the pre- and the post-survey (first two columns: PreMean and PostMean) as well as significance levels (last column).
Figure 10. Pre and post comparison of means on different evaluative dimensions.
Differentiated by evaluation dimension, both significant differences as well as no significant differences can be noted. In detail, it is particularly scientific skills, reported and tested knowledge where participants increase their scores and remain still at a higher level even after one month of participation. Their attitude towards science, their motivation to join, and their perceived self-efficacy does not change significantly.
In the following we will investigate more details on subscale level.
Attitude towards science: This subscale comprises nine items, with statements such as “I am interested in learning more about particle physics” or “I enjoy reading about science related topics” (See complete questionnaire in annex).
As the pre-mean score of 4.29 shows, the attitude towards science in general was already rather high and did not change over the course of participation.
Motivation to join: This subscale consists of twelve items and comprises items covering intrinsic and extrinsic motivation. Intrinsic motivation describes an inherent satisfaction for a certain activity, while extrinsic motivation describes a behaviour determined by external rewards or punishments (Ryan & Deci, 2000). Example of items are: “Because I think it’s a good thing to do” (intrinsic), “Because I believe in can contribute to scientific research” (extrinsic) and “For the recognition I get from others” (extrinsic).
As the means show, the motivation to join is on average on a medium level, with a score of 3.58, which remains at the same level. A detailed analysis on item level shows that this is mostly because of intrinsic motivation and not because of reasons speaking for extrinsic motivation. For instance, participants decided to join because they wanted to spend their spare time doing something useful (item 3b.3, see Annex), or because they enjoyed getting involved in scientific activities (item 3b.4), both hinting towards intrinsic motivation; and less because they wanted to gain recognition by others (item 3.9) or because they wanted to connect with their professional activities (item 3b.6), i.e. extrinsic motivation factors.
Science Skills: The question block on science skills comprised five statements referring to the citizen science project (e.g. ‘I know how to categorise the data in the Deep Sea Explorers project”). As the paired sample T-test shows, there is a significant increase in science skills from the pre- to the post-survey. In other words, participants, according to their own ratings, gained scientific skills over the course of their participation in the Zooniverse projects.
Self-efficacy: Four items related to self-efficacy in doing science, i.e. believing in one’s own skills in science-related activities, included statements such as ‘I think I am pretty good at following instructions for scientific activities’ and ‘It takes me a long time to understand how to do scientific activities’. The perceived self-efficacy in science-related skills did not improve significantly and was at both points in time at a medium level, with scores around 2.3.
Knowledge: The knowledge block was divided into reported knowledge and tested knowledge, which allowed for a comparison of an objective and subjective level of knowledge in the respective field. In the reported knowledge section (nine items), participants were asked whether they felt confident explaining specific scientific terms, while in the tested knowledge part they had to identify which items were correct and which were incorrect. These ‘objective’ items comprised statements regarding the purpose of research infrastructures and some statements referring to the specific scientific fields of the individual citizen science project.
Demographic analysis of score changes
In the following, the pre- and post-test results are contrasted against the demographic characteristics of the participants, allowing for an analysis from a democratisation perspective.
Gender differences: Due to the small proportion of people who declared themselves as non-binary and people who preferred not to say (c.f. demographic analysis), the data associated with these were omitted from the following analysis.
No significant differences were to be found between females and males in their Zooniverse engagement in terms of number of classifications, time spent per classification and overall time.
In the pre- and post-questionnaires there are some, albeit marginal, gender differences (cf. Figure 11). Males show a more positive attitude toward science (pre), and higher levels of science-related skills (pre and post), and self-reported knowledge (pre and post) and tested knowledge (pre). These differences, with a few exceptions, were no longer found in the post test. In other words, some of the marginal gender gaps were closed in the post-test with respect to attitudes towards science, as well as tested knowledge, while others persisted (science-related skills and self-reported knowledge).
Figure 11. Pre and post comparison of means on different evaluative dimensions by gender.
Age differences (Overview in Figure 12): Attitude towards science was already high among all age groups in the pre-test and did not change over time. The motivation to join was slightly more pronounced among the youngest age group (compared to the oldest) both in the pre- and post-test. Perceived science-related skills were also higher among younger age groups than older ages, again at both points in time, while science-related self-efficacy did not vary among the different age groups. While in the pre-test it was the older age group who felt particularly more confident in explaining scientific terms, younger age groups did catch up over time.
Figure 12. Pre and post comparison of means on different evaluative dimensions by age groups.
Differences by professional scientific background (Overview in Figure 13): Most differences appeared between people with scientific backgrounds and people without scientific backgrounds.
Figure 13. Pre and post comparison of means on different evaluative dimensions by professional background.
The attitude towards science was more positive among people with a scientific background (pre and post), and so was the perceived self-efficacy (pre and post), while science-related skills differed significantly only in the pre-test and the gap was no longer there in the post-test. Reported scientific knowledge persisted among people with a scientific background (i.e. pre and post) who rated their knowledge with regards to specific terms higher. Interestingly, there were no differences in the tested knowledge and motivation to join.
Differences by engagement level: When compared against engagement level (composite indicator of number of categorisations plus total classification time), the only differences appear in the post-questionnaire. Highly engaged people reported more knowledge and a higher motivation to join.
Differences by visual impairment: No significant differences resulted from a comparison of people who reported a visual impairment with people who did not.
Discriminated against group (Overview in Figure 14): This differentiated according to membership of a discriminated against group (for different reasons, such as gender, sexual orientation, race, migration background, etc) with only one marginal, but significant, difference being apparent, as the following overview shows. The motivation to join was significantly higher for people who felt discriminated against both in the pre as well as in the post test, which is quite remarkable. We can hypothesise that they valued the option to participate in such a study more because they may otherwise experience fewer opportunities to do so.
Figure 14. Pre and post comparison of means on different evaluative dimensions by discrimination experiences.
Discussion and conclusion
Citizen science projects face a critical challenge in establishing a unified evaluation standard, hindering effective cross-project comparisons and comprehensive indicators for democratisation in citizen science projects (Bonney et al., 2009; Bonney et al., 2014; Jordan et al., 2015). Whether citizen science projects are successful in attracting citizens beyond the ‘usual suspects’ is essential to showcase the level of inclusion and accessibility of the project. Our study seeks to address this gap by operationalising the gains of citizen scientists from a democratisation perspective. Furthermore, with our study design we attempt to rectify the limitations of prior studies, such as retrospective analysis, potentially advancing the evolution of evaluation standards in citizen science. However, acknowledging certain drawbacks is essential. The research design's limitations stem from the absence of a control group, making it challenging to attribute observed effects solely to participation. Desirability effects and the self-selection bias of highly motivated participants pose potential concerns. Despite these challenges, our study incorporates objective knowledge measures, bridging the gap between subjective and objective indicators. The divergent procedures for collecting pre- and post-questionnaire data, though unavoidable due to technical constraints as described above, may have influenced post-questionnaire response rates. The procedure for collecting the information in the pre and post questionnaires was not the same. The fact that the latter was accessed via an email link might have resulted in fewer responses in the post-questionnaire as it implies several hurdles (e.g. changed email address, wrong email address, email in junk mail etc.) compared to a link that can be accessed directly within the project that citizen scientists were working on, i.e. the Zooniverse projects.
In light of the results and their limitations, our investigation provides insights into participant benefits and demographic characteristics in citizen science projects.
The first research question reveals positive outcomes, indicating gains in scientific skills and knowledge attributable to participation. While the motivation and attitude toward science remained consistent, our findings align with previous studies, emphasising the transformative potential of citizen science in terms of scientific skill and knowledge gains, even in contributory-focused projects. Obviously, it cannot be ruled out that these gains have been caused by other not-controllable factors. The motivation to join, the attitude towards science and the level of self-efficacy in the science domain does not change over time. This result is similar to other studies (Bela et al., 2016; Richter et al., 2016; Riesch & Potter, 2014; Stepenuck & Green, 2015), which also detected knowledge gains and an evolution of scientific skills. It is remarkable that the citizen science projects that are mostly contributory in nature without a deeper involvement of citizen scientists in different phases of the research process, and thus not in line with the extreme citizen science approach, nevertheless led to an increase in the mentioned dimensions. Even a citizen science project with lower levels of involvement as in this study can have positive impacts on knowledge acquisition and the development of scientific skills.
While skills and knowledge improved, other dimensions did not change. The attitude towards science was already high to begin with, not leaving much room for change. On an item-level, we recognise similar motivational drives as have been reported in other studies (Land-Zandstra et al., 2016; Martin et al., 2016; Silva et al., 2016). People join because they have the desire to contribute to a “greater good,” thus merely for intrinsic and not extrinsic motivation. The desire to contribute to the objectives of an important or interesting project is an attraction factor for citizens and explains why they join the project in the first place. For this reason, the communication of the project’s mission, achievements and the scientific contributions of the individual citizen science projects is key in recruiting new volunteers and keeping them involved in the project activities.
The fact that the attitude towards science was already positive in the beginning indicates, however, that probably people with negative attitudes towards science are difficult to attract to citizen science projects.
The second research question explores inclusivity and accessibility. Despite a skewed participation pattern towards males with scientific backgrounds and higher education degrees, the projects exhibit inclusivity by engaging participants across genders, ages, and educational backgrounds. Notably, individuals with discrimination experiences are particularly motivated to participate, highlighting the potential of citizen science in empowering marginalised groups. In contrast to the idea of democratisation of access to research and the production of scientific knowledge for all, the participation pattern often to be found in citizen science projects is males with a scientific background and higher educational degrees. Thus, it is important to counteract this pattern of involving solely the ‘usual suspects’ and to analyse the degree of inclusivity and accessibility in citizen science projects. The analysis of the demographic characteristics shows a slight over-representation of males compared to female participants, by more than 10%, and a sample skewed towards higher education degrees. However, also participants with lower educational degrees were able to access and contribute to the projects. In terms of age, we find quite a balanced sample, with all age groups represented, from the very young, below the age of 20, to elderly people above the age of 80. The participation of elderly people might be attributed to the engagement of the research teams in lectures and courses with the elderly in line with the engagement strategy of the projects. A minority have a background in science, which again indicates that the citizen science projects have successfully attracted people beyond the ‘usual suspects’ (Curtis, 2018). This is further confirmed by the fact that a considerably high proportion of participants feel that they are members of a discriminated against group and a few indicate a visual impairment speaking also for the technical accessibility of the projects. The demographic analysis of the data reveals that people with discrimination experience are particularly motivated to participate in citizen science projects. Taking into account demographic dimensions, it was not possible to identify any groups of people as having benefited less in terms of knowledge acquisition and development of scientific skills.
Our study suggests that citizen science projects, even with limited citizen involvement, can positively impact knowledge acquisition and scientific skill development. The importance of engagement strategies and accessibility efforts is underscored, emphasising the need for a more diverse participant base, including females, individuals with disabilities, and those with lower educational backgrounds.
Future research should delve into sustained participant impacts, refine evaluation methods, and explore demographic influences further. Emphasising engagement strategies and accessibility testing can foster greater inclusivity, fulfilling the democratising potential of citizen science. Understanding the nuanced gains and influencing factors is crucial for advancing the democratisation of scientific engagement in citizen science projects.
Ethics and consent
The consent of participants was given online on the Zooniverse platform. For the analysis of the usage data from Zooniverse (https://www.zooniverse.org/), the REINFORCE participants were informed and protected by the privacy policy statement published on the Zooniverse platform (https://www.zooniverse.org/privacy), which defines the management and handling of the user data. To register on the platform, the REINFORCE participants had to confirm that they agreed with the policy and they were additionally explicitly asked for their consent allowing their user data collected by Zooniverse to be shared with the REINFORCE project. According to the REINFORCE ethical guidelines, only relevant and necessary data was collected, stored, and analysed.
The Reinforce project underwent an ethics check during grant preparation and as a result dedicated a Work Package to ethics requirements with respective reports and detailed description of ethics processes (POPD, and H Requirement). Additionally, the project provided an ethics handbook comprising a project information sheet, consent sheets, and detailed description of all ethics procedures. The ethics procedures were positively evaluated during project monitoring.
