Plain language summary
This paper describes how a new e-health training was developed, of which the aim is to teach young people about emotions using scientific theory. Topics in the training include: ‘What is an emotion?’, and ‘How can we recognize emotions?’. Learning about emotions can help youth understand their emotional lives, which can promote well-being and prevent psychological issues. The training was developed to be a scientifically grounded training that is engaging for users. We built this training for a non-clinical young audience (16–22), as this is a critical developmental life stage where having access to mental health knowledge is important. The training was built using a theory of emotion that conceptualizes emotions as reactions to a goal-relevant event and consisting of different components: thoughts, bodily sensations, gestures (expressions), actions one wants to take and subjective feelings. Feedback from youth was collected through focus groups, sessions of the Youth Advisory Board, and an anonymous survey. From their input we learned that young people find the following important in digital mental health trainings: personalization, relatability, fast and colorful graphics, professional audiovisual elements, interactivity, ease-of-use, privacy protection and scientific legitimacy. The resulting training consists of psychoeducation videos and content related questions, stimuli from which emotions are identified (i.e., photo and video expressions, and emotion vignettes) and reflection exercises. The content was organized in 10 ‘sessions’ and these were categorized into four ‘challenges’: introduction, parts of an emotion, regulating emotions, recognizing emotions and a summary session. The sessions increase in difficulty and unlock as the user progresses. To conclude, we developed a new e-health training about emotions that is both scientifically grounded and takes the end-user into account.
Introduction
Our emotions serve as powerful tools for navigating our social world. Emotions are evolved mechanisms that can signal when to avoid (anxiety), when to be cautious (fear), when to affiliate (joy, love), and when to retreat and rest (sadness; Frijda & Mesquita, 1994; Keltner & Gross, 1999; Lerner & Keltner, 2001; Shiota et al., 2004). These natural processes can go awry when signals are misinterpreted, or when reactions to these signals are misdirected or unregulated (Gross, 2008). Identifying and understanding our emotions and the underlying processes from which they unfold has important implications on our functioning (Castro et al., 2016; Trentacosta & Fine, 2010). Additionally, such skills can be taught and learned (Hodzic et al., 2018; Izard et al., 2011; Trentacosta & Schultz, 2015), and digital psychoeducation programs have the potential to provide this type of training efficiently (Schlegel, 2021). The term ‘emotion competence’ (EC) has been used to describe the skill-based capacity to identify, understand, correctly use and regulate, or manage one’s own and others’ emotions (Fontaine, 2016). Emotion competence (or ‘emotional intelligence’ as it is also often referred to) has been linked to positive life outcomes, including health, well-being, and job-related success (e.g., Brackett et al., 2011; Petrides et al., 2016).
Deficits in emotion competence have been consistently found to be both a risk factor and core feature of psychopathology. For example, researchers have found emotion dysregulation (e.g., the excessive use of rumination or suppression) as well as deficits in the ability to accurately recognize other people’s emotions to be related to mental health disorders such as bipolar disorder (Glenn & Klonsky, 2009), anxiety, depression, aggression, and eating pathology (Brockmeyer et al., 2016; Dalili et al., 2015; McLaughlin et al., 2011). Mental disorders account for a large proportion of the disease burden in young people around the world (Patel et al., 2007; Vigo et al., 2018) and there is a global trend of earlier onset of mental disorders (Marcus et al., 2012; Patton et al., 2016). The incidence of anxiety and depression markedly increases starting mid-adolescence and peaks during the sensitive period of young adulthood (Kessler et al., 2005). Moreover, the adolescent and young adult years are central in the development of capabilities related to health and well-being, such as EC (Halberstadt et al., 2001; Patton et al. 2016). Yet, despite the need for mental health training and interventions, a large majority of young people do not receive or have access to them (McGorry et al., 2013; Thornicroft, 2008). As a result, interest in mental health smartphone apps has surged as more people are seeking digital tools for self-improvement (Torous et al., 2018). Given the importance of EC in adolescence and young adulthood, training these competencies via a digital platform can be an impactful preventative measure.
Digital interventions
Training emotional competence skills via a digital platform, e.g., as part of a mental health training app, has the potential to provide an effective and scalable way to improve well-being in the younger population (Newbold et al., 2020). Digital mental health trainings are highly accessible, cost-effective, can receive high acceptability by youth, and can improve mental health outcomes (Lattie et al., 2019; Price et al., 2014). Furthermore, accessing mental health trainings digitally can allow young people to engage with their emotional lives in a confidential space outside of any stigma or discomfort linked to discussing their mental health (Levin et al., 2018). Thus, mental health apps may engage young people who typically would not seek help through traditional routes.
Despite this tremendous potential and the increasing interest of health apps in the general public and scientific communities, there are two major issues that mental health apps face. First, the majority of mental health apps available for download are not theoretically grounded nor guided by scientific evidence during app development (Mani et al., 2015; Neary & Schuellen, 2018; Nicholas et al., 2015). For example, in a review of anxiety management apps, 67% of apps lacked any involvement from mental health professionals in their development (Sucala et al., 2017). Moreover, many apps do not include the type of content that has been shown to improve mental health outcomes. To assess this, Wasil & colleagues (2019) coded 27 apps for empirically supported treatment elements. They found that although some popular treatment elements for depression and anxiety were common across apps (e.g., meditation, relaxation), they found that other core treatment elements were rarely included (e.g., exposure, restructuring). Kertz & colleagues (2017) also conducted a review of worry and anxiety apps and found that only a quarter of the apps contained content that was consistent with evidence-based strategies for generalized anxiety disorder. Such findings suggest that there is a lack of scientific and clinical expertise during app development.
The second issue that mental apps face is that rates of app engagement are often surprisingly low. In a market study that measured app retention of 37,000 apps, it was found that 63% of users typically visit an app less than 10 times (Localytics, 2017). Ensuring app retention is particularly important in mental health apps, as loss of motivation is a common symptom of clinical and subclinical mental health issues (Dickson et al., 2017; Winch et al., 2015). One study found that only 14% of people who downloaded a Post-Traumatic Stress Disorder (PTSD) app used the app the day after they downloaded it (Owen et al., 2015).
Researchers have identified poor usability and lack of user centric design as key factors that contribute to the low engagement rates of mental health apps. In a clinical review of user engagement in mental health apps, Torous & colleagues (2018) identified several themes underlying poor engagement of mental health apps. These included poor usability (apps were not designed with users in mind), distrust from users regarding privacy and scientific merit, and apps not solving problems users care about. In another study that asked about mental health app preferences, participants highlighted visual appeal, intuitiveness, simplicity in language and ease of use to be important preferences and were more likely to adopt an app if it was backed by research (Schueller et al., 2018).
Hence, involvement of end-users in the design and development of apps has been identified as a crucial step in improving usability and adoption of mental health apps (Coyle et al., 2007). Particularly in the mental health space, researchers have highlighted the importance of including young people in the development and design of an app in order to better match the needs and preferences of these users (Hetrick et al., 2018).
Reflecting on these recommendations, we built an emotion knowledge skills training for youth that is firmly rooted in scientific theory and developed with the cooperation and input from end-users. This training is a part of a larger European research project ‘Assessing and Enhancing Emotional Competence for Well-Being (ECoWeB) in the Young: A principled, evidence-based, mobile-health approach to prevent mental disorders and promote mental well-being’ (Horizon, 2020 Grant 754657; Newbold et al., 2020). The aim of the ECoWeB project is to build and test the ‘MyMoodCoach’ app1 which includes various training programs in emotion competence, one of which is the training described in this paper.
The challenge of creating a scientifically based training that is attractive and engaging to a young audience was taken on by combining a strong theoretical basis and including a diverse sample of young people in all stages of the process. This paper aims to describe the processes and learnings of building and testing a digital emotion skills training.
Theoretical background
As part of the ECoWeB project, we aimed to build a training focused on Emotion Knowledge as it is conceptualized by Scherer (2007) and Fontaine (2016). The contents of the new training adopt a well-developed ability model, the Emotional Competence Process model (ECP; Scherer, 2007) that draws on established findings about the mechanisms underlying emotional functioning (Mehu & Scherer, 2015). The ECP distinguishes various emotion competence abilities which are implicated in mental well-being in health, such as awareness, recognition, regulation, and understanding of emotion processes. More specifically, emotion knowledge involves the awareness of the personal and situational factors that determine the elicitation of a specific emotion, accurate recognition and perception of one’s own emotions and the emotions expressed by others, empathic understanding and perspective taking for the emotional reactions of different people, and understanding emotion dynamics.
The ECP is based on an empirically validated theoretical process model of emotion called the Component Process Model (CPM; Scherer, 2009) which regards emotions as a dynamic process triggered when an individual encounters a goal-relevant event and that is set in motion by (low level) cognitive appraisals. The CPM is an emotion theory within the componential emotion approach, which regards emotions as a process that includes five components: appraisals (of the event), bodily reactions, expressions, action tendencies, and subjective feelings (Scherer, 2009; Fontaine et al., 2013). Appraisals are cognitive evaluations of the event or object (e.g., that the event is controllable, predictable, violates one’s norms). Bodily reactions refer to physiological changes that prepare individuals to respond to the emotion-evoking event (e.g., increased heart rate). Action tendencies are behavioral inclinations or motivational forces (e.g., wanting to hide). Expressions include observable facial, postural or vocal changes (e.g., frowned eyebrows). Subjective feelings refer to the conscious awareness of the emotion which allows for verbal communication about the emotion process. This componential approach to emotions has been supported by cross-cultural psycholinguistic research using 34 samples from 27 countries in 23 languages that consistently revealed that emotion words refer to changes in all five components (Fontaine et al., 2013).
When applying the componential emotion approach to ability-based tasks, we operationalize emotion understanding as understanding the probable presence of the five emotion components in reaction to a relevant event, emotion identification as the ability to correctly identify an emotion process using information from all of the five emotion components, and emotion regulation knowledge as knowing how to modify one or more of these five components (Fontaine, 2016).
Basing the emotion knowledge training on this combined set of competencies has several strengths. First, using a model that is based on normal functioning allows for creating content that can be useful for preventive and psychoeducational training for people regardless of any symptomology. Other mental health trainings are typically derived from clinical models and are targeted to those with known mental health symptoms (Parker et al., 2018). Second, focusing on emotional processes that can be trained through standardized tasks allows us to later scientifically assess the efficacy of the training with ability-based assessment tools. Lastly, standardized training material benefits from being more easily scaled-up in digital formats and transferred to other languages.
Objective
To summarize, this paper describes the development of an emotion knowledge training that was designed to be delivered through the MyMoodCoach app for youth aged 16–22 within the ECoWeB research project. To avoid issues related to low engagement, we used an iterative approach (cf., section Study design) and involved youth in all stages of development. We aimed to build a training that was both firmly embedded in theory and engaging to a non-clinical young population.
Methods
Study design
We adopted an iterative approach in which the training content and design were developed incrementally and adapted over the course of the project. This allowed us to make changes at various steps of the process and be responsive to user feedback. It also allowed us to negotiate between ideals and the technological and ethical practicalities we faced. Training design elements were thus adapted to meet limitations of programming resources. In Table 1, characteristics of the focus group meetings are listed. In Table 2, characteristics of the Youth Advisory Board (YAB) meetings are listed.
Table 1. Characteristics of the focus group meetings with A, B, C referring to each of the three master student researchers.
Focus group
date | Student | Sample
size | Gender distribution
participants | Age group
participants | Ethnic
origin | Main topic |
|---|
| 24/07/2018 | A | 5 | 100% female | 100% +18 | Belgian | Script session 1
Emotion
exercises |
| 25/07/2018 | B | 4 | 100% female | 25% +18 | Belgian | Script session 1
Emotion
exercises |
| 12/08/2018 | A | 8 | 100% female | 50% -18 | Belgian | Voice recording
tool
Mood Tracking
Video Styles |
| 18/08/2018 | C | 3 | 66%
female | 100% +18 | Turkish | Script session 1
Emotion
exercises |
| 29/08/2018 | C | 4 | 50%
female | 100% +18 | Turkish | Voice recording
tool
Mood Tracking
Video Styles |
| 30/08/2018 | B | 5 | 100% female | 40% +18 | Belgian | Voice recording
tool
Mood Tracking
Video Styles |
Table 2. Characteristics of the Youth Advisory Board (YAB) meetings.
| YAB Date | Researcher | Sample
size | Gender distribution
participants | Age group
participants | Ethnic
origin | Themes |
|---|
| 25/09/2018 | 2nd author | 5 | 60% female | 60% 18+ | Belgian | Graphics & avatar
Gamification
Target population &
promotion app
Emotion Knowledge
Training
Emotion Analyst
Voice Recording |
| 29/04/2019 | 2nd author | 6 | 66% female | 100% 18+ | Belgian | App mock up |
| 26/02/2020 | 2nd author | 2 | 50% female | 100% 18+ | Belgian | Evaluation after test
round of app |
Focus groups
Participants. Data from six focus groups (N = 29, unique participants N = 25) that were organized to build and test content during all stages of training development (i.e., deciding on the structure and framework of the training, making psycho-education scripts and animation videos, selecting stimuli, creating scenarios, listing response options, and refining materials) have been used. Detailed information regarding the focus group meetings is listed in Table 1. These focus groups were led by three female master’s students (referred to in Table 1 as A, B and C). The age of participants we were looking for was 16 to 21 years old. Special focus was placed on recruiting participants from minority groups, as these young people tend to have even more issues with accessing mental health than in general (e.g., Lee-Tauler et al., 2018). Specifically, we chose to recruit participants of Turkish origin as this is one of the largest minority groups in Belgium (cf., Wets, 2006). The master’s students were instructed to recruit participants through their personal connections and social media. When a potential participant showed interest, they contacted the master student, after which the student emailed them more information, and asked about their age. This allowed the students to provide the passive parental consent in advance (cf., procedure). Participants could join multiple meetings or just one, and each meeting was open to new participants (there was a maximum of 10 participants per meeting). For student A, in the second meeting no participants from the first meeting returned. For student B, in the second meeting three participants from the first meeting returned, and two new participants joined. For student C, in the second meeting, one participant from the first meeting returned, and three new participants joined. No information on the relationship between the students and their participants was collected by the researchers. The authors of this paper did not establish a relationship with participants prior to study commencement. Participants were aware of the general goal of the focus group, namely, to collect feedback and opinions about training material and other general questions regarding the MyMoodCoach app. The message towards the focus groups was that the researchers were open for suggestions and improvements, creating an open atmosphere.
Procedure. The authors were already collaborating with master’s students regarding the ECoWeB project due to the link of their master’s thesis dissertations with the project. The students were informed, guided, and instructed by the authors to organize and run the focus group meetings. In addition, intermediate meetings between the students and the authors took place to closely follow up the course of events. The authors provided a script to the students which included information about the general goal of the larger project (i.e., ‘improve emotional competencies - the capabilities to understand, recognize and regulate emotions in ourselves and others’ and ‘building an app made for young adults to help them learn about emotions’) and the goal of the focus groups. The script also mentioned that participation is completely voluntary, participants can choose whether or not they answer questions and can leave at anytime. Participants were informed that the sessions were not meant to ask them about their personal emotions, that there were no correct or incorrect answers or any type of evaluation. It was also explained that participants of the focus groups need to keep the content of the focus groups confidential.
The master students were free to find a location (a formal room, not an informal location like a coffee bar etc.) or could ask the authors for a meeting room. After the general introduction, all participants received an Informed Consent form to sign before the meeting continued. The students handed in all informed consent forms to one of the authors after their meeting had taken place. Before the meeting took place, all participants younger than 18 were instructed by the students to inform their parent(s) about their wish to participate in the focus groups. The parents could sign and return a passive parental consent to let the students know in case they did not agree with their child’s participation. In the Informed Consent form the statement ‘I agree that the researchers will use my replies for further developments in the ECoWeB project’ was included to clearly indicate the purpose of the collected data.
The focus group meetings took approximately one to two hours, during which the students took notes to use later for the minutes of the meeting. The participants could also take notes on the sheets with the assignments and were asked to not write their identification information on these. Students then wrote a summary of their focus group which was used by the first author to learn more about the training materials and the MyMoodCoach app for further improvements. All handwritten notes were either destroyed or submitted to one of the authors. The minutes were not returned to the participants for feedback or comments. The authors decided that discussing each focus group topic across three sessions would be sufficient to obtain reliable findings.
After focus group meetings ended, each participant received five euros. As advised by Seymour (2012), the amount of payment was carefully chosen to make sure it is not disproportional to the investment of participants, it is not coercing participation, and it does not influence the decision regarding risk taking as this participation is not likely to include any risks. This payment is not expected to decrease intrinsic motivation as five euros might be considered not worthwhile for participation. We assume this incentive is merely seen as a bonus for participation and possibly experienced as a form of appreciation (cf., Resnik, 2015).
Youth Advisory Board (YAB)
Participants. Five meetings with the Youth Advisory Board (YAB) took place (N= 25; unique participants N=12). Data from three of those sessions are within the scope of this paper and were included in analyses. Detailed information regarding these meetings is listed in Table 2. The second author of this study organized and chaired these meetings. The age of participants we were looking for was 16 to 22 years2. The second author, who has a master’s in clinical psychology, was inexperienced in leading focus group meetings and hence used guidelines provided by ECoWeB team members and the first author. Guidance by ECoWeB team members was provided in the format of a PowerPoint presentation which included information about the project, the goal of the YAB meetings and the questions for the group. Recruitment took place through social media, the Ghent University Psychology department, and personal connections. Four out of the twelve participants were acquainted with the researcher through different contexts before the study and were encouraged to reply as openly as possible. When a potential participant showed interest, they needed to contact the second author. Following that, the second author provided more information and asked their age, after which a passive parental consent form was sent to this person in case they were younger than 18. Participants of the YAB were encouraged to come to several meetings. There were seven participants who came to more than one of the YAB meetings and five participants who joined only one meeting. Participants were aware that the second author was also a researcher in this project, and thus co-developer of the materials. The second author emphasized the importance of both positive and negative feedback.
Procedure. Participants in the YABs received, at the beginning of the first meeting, information regarding the ECoWeB project and the goals of the YAB. The participants were informed that no personal nor sensitive information would be sought. Participants could ask for more information and, after the introduction, they were invited to read and sign an Informed Consent form. For participants younger than 18 years, in addition to their own signed informed consent, passive parental consent was sought before the meeting. The parents could sign and return a passive parental consent to let the researcher know in case they did not agree with their child’s participation. In the Informed Consent form, the statement ‘I agree that the researchers will use my replies for further developments in the ECoWeB project’ was included to clearly indicate the purpose of the collected data. An optional section in the Informed Consent form was regarding the audio-recording of the meeting, of which no participants opted out. The second author took notes during the meeting. After the meeting, these notes were digitalized in a summary and complemented with notes from the audio-recordings to form a report. The participants could also take notes on the sheets with the assignments and were asked to not write their identification information on these. The audio-recordings were deleted immediately after the reports were written. The reports were used by the first author to learn more about the training materials and the MyMoodCoach app for further improvements. The YAB summaries have also been taken into account in the larger ECoWeB project. The notes and reports were not returned to the participants for feedback or comments. Researchers decided that investigating each topic in one national YAB session would be sufficient to obtain reliable findings.
The YAB meetings took place in a faculty meeting room with only the researcher(s) and YAB participants present. The approximate duration of YAB meetings was two hours and participants received 20 euros for each session. For the Youth Advisory Boards, the incentives were similar across the different European countries in the ECoWeB study. After reflecting within the large consortium, it was decided that a reward of 20 euros was appropriate for this situation and the low level of risk since participants were not expected to share sensitive personal information, only their opinions regarding suggestions we make in the project (cf., Resnik, 2015; Seymour, 2012).
Pilot study
Participants. We collaborated with the Ghent University course ‘Coaching Skills’ for the pilot study of the newly developed Emotion Knowledge training. Ghent University students were chosen since this research is conducted by Ghent University researchers. This course was an ideal choice because emotional intelligence is a topic included in this course. The students who were enrolled in the ‘Coaching Skills’ course (N=94) completed the new Emotion Knowledge training in a web-based format as part of their course requirements. This web-based format was programmed by our team using Qualtrics. Another reason to collect feedback from students in this course is because they were accustomed to reflecting upon ways to educate people. Therefore, it was especially interesting to collect their insights about our developed training. Recruitment was done by inviting these students through the general student communication platform (Minerva) to complete an additional optional survey which was programmed in Qualtrics. A total of 17 people completed this additional optional survey. A copy of the survey can be found in the Extended data (Hovasapian et al., 2022).
Procedure. Both the students and the instructors from this course received information on how the training would be accessible and what was expected from the students. Here we do not provide further details on the procedure for completing the training as these data are not included in the current study. The included data only refers to the voluntarily given feedback on an early version of the training. The feedback questions were the following: ‘Overall, how did you feel about the difficulty of the questions?’, ‘Please answer the following questions about the videos you watched at the beginning of each session. - What did you like about these videos? - What did you not like about these videos?’ ‘These questions are about the two-answer questions that were asked after each video (e.g., "being satisfied with the current situation is an example of which emotion?") - What did you like about these questions? - What did you not like about these questions?’, ‘These questions are about the figures where you had to guess which emotion was depicted. - What did you like about these questions? - What did you not like about these questions?’, and ‘These questions are about the reflection questions where you each had to give an answer about your experiences: - What did you like about these questions? - What did you not like about these questions?’. Students were informed that their feedback would be used to further improve the training.
Materials
Stimuli to create emotion recognition exercises included photos and videos from validated databases which were freely available to investigators for use in scientific research. For photos, we used the Amsterdam Dynamic Facial Expression Set (ADFES; van der Schalk et al., 2011), which includes a standardized set of expressions for nine discrete emotions (joy, anger, fear, sadness, surprise, disgust, contempt, pride, and embarrassment). The set includes 10 male and 12 female models with 10 of them of Mediterranean descent and 12 of North European descent. Models were trained to express prototypical elements of emotion displays based on a body of research that links facial micro-expressions (called Action Units) to specific emotions (e.g., Ekman & Friesen, 1978; Keltner, 1995; Tracy & Robins, 2004).
For videos, we used the Geneva Multimodal Emotion Portrayals database, which includes portrayals of 18 emotions in different intensities (GEMEP; Bänziger et al., 2012). Twelve emotions had been selected to evenly cover four quadrants of the emotional valence–arousal space: joy, amusement, pride—high arousal/positive valence; pleasure, relief, interest—low arousal/positive valence; anger, fear, despair—high arousal/negative valence; and irritation, anxiety, sadness—low arousal/negative valence. Actors were videotaped from their upper torso upward to display facial and postural emotion cues. They pronounced a sentence that did not have semantic meaning so that emotion cues could still be picked up by the voice, but not by specific words. Actors were instructed by a professional director to remember real-life scenarios related to each emotion to maintain authenticity of the expression.
Written emotion identification scenarios were collected and created to include in the training. In addition to the scenarios created by youth in focus groups, we collected scenarios from an existing data set of participants who filled out the Geneva Emotion Analyst, during which participants described an emotion experience they had (Scherer & Meuleman, 2013).
Training development process
The development of the training followed three general steps and included focus groups, a Youth Advisory Board and a pilot study. Table 3 lists examples of questions asked in focus group and YAB meetings throughout each of these steps.
Table 3. Select questions from focus group and youth advisory boards.
| Category | Prompts |
|---|
| Recruitment | What do you think is the best way to let a young person know about a well-being app? |
| What information would you want to know about it before using it? |
| Which types of advertising would work best with people in your age group? |
| What are your general impressions of the recruitment materials? |
| Branding | What could we call the app to make it appealing to young people? |
| Who would be interested in an app like this? |
| Motivation | Are there any well-being or mental health apps you currently use? |
| What do you like about those apps?/ What do you dislike? |
| What would make a mental health app attractive to you? |
| Barriers | What barriers or pain points would stop you from using a mental health app? |
Feedback on training
content | What are your general impressions of the training content? |
| What do you like/not like about the exercises? |
Voice recording and
sensor usage | Would you agree to give an app permission to store or record voice data if it was secure? |
| What do you think about sensors on a phone being used to help track activity
anonymously? |
| Emotion tracking | What do you think about a tool that allows you to visualize your tracked emotions? |
| If you were using mood tracker app, how many times a day/week would you complete the
tracker? |
Audiovisual
preferences | What did you like/not like about this specific video style? |
| What are your general impressions of the graphics and avatar presented here? |
| Gamification | What are your general impressions of using gamification in an app like this? |
| Do you want to be able to access all features or is it ok to have parts that unlock with
usage? |
| Usability | What features of an app would encourage usage? |
| What are your general impressions of receiving notifications? |
| How long did you use the app when you used it? At what point did you close the app? |
Step 1: Training design and framework. The first step consisted of a search for theoretical frameworks and information from existing data in the creation of relatable content and scenarios for youth. We reviewed scientific models of emotion, namely, the Emotional Competence Process model (ECP; Scherer, 2007) and the Component Process Model (CPM; Scherer, 2009). Through our conversations within the ECoWeB consortium, we also considered how gamification theory (e.g. Zichermann & Cunningham, 2011) and user experience concepts (Pernice, 2016) can inform the design and testing of the training.
In line with the Emotional Competence Process model (ECP; Scherer, 2007), the focus of the training was to develop the skills related to the ability to perceive, understand, predict, and manage emotion processes in the self and in others. The aim was to develop a psychoeducation for young people about these different facets of the emotional experience using a strong conceptual model. By doing this, we advance the state-of-the-art by training emotion competence skills directly on a digital platform. Through this training, participants would learn to identify emotions from cues from each of the emotion components, to understand the interplay between the emotion components, and to understand how an emotion can be regulated by changing emotion components. To gather this information, we drew on empirical data from the GRID studies (provided to us by Klaus Scherer and Johnny Fontaine), emotion theory (cf., Fontaine et al., 2013; Fontaine, 2016), gamification mechanics (Zichermann & Cunningham, 2011), advice from authors and collaborators from the ECoWeB consortium. We also obtained feedback from participants about their experiences with other mental health apps, hooks and barriers to mental health app usage, as well as their thoughts on recruitment, branding, and usability of apps.
Step 2: Content development. Following an iterative approach, we broke up content development into several steps and involved participant feedback in the created material. Several focus groups were conducted to provide feedback. These focus groups were held at the early stages of content development (cf., Table 1). Participants helped build content by providing scenarios that youth could relate to, reviewing psychoeducation video scripts, and answering questions related to what youth sought in well-being apps and the general perceptions to mHealth technology and privacy.
A Youth Advisory Board (YAB) met several times at different stages of the project (cf., Table 2). This group of participants was deeply involved and informed about the nature of the project and gave both broad level and specific feedback about design, content, and direction. At the earliest stages, participants in YAB meetings, as well as those in focus groups, commented on their use and interest in mental health apps, their motivations and barriers, topics of concern, and their interest in specific types of functionalities. This helped inform the design of training content. They also provided feedback on training design, earliest versions of psychoeducation video scripts, and video animation styles. Participants also took part in the development of the training itself, such as providing scenarios from daily life to include in emotion identification exercises.
Step 3: Pilot testing. The first phase of the full content of the training was programmed into Qualtrics (Table 4) to pilot test the content in terms of understandability, relatability and to determine duration of each session. Collecting data on how long it took for participants to fill in would allow us to estimate our expected dosage expectations. In this pilot study, participants completed a draft of the full training content before it was integrated into the app. Programming into Qualtrics allowed us to develop and edit content before the app platform was ready. Such low-fidelity prototyping is a recommended user research practice that involves testing basic models or examples of a product rather than testing on the full version of an app so that changes can be made more quickly, iteratively, and with fewer loss of resources (Pernice, 2016). Testing our training in Qualtrics reduced the number of edits that were done after it was already configured into the app. Participants completed all parts of the training and were subsequently asked if they wanted to participate in an online survey about their experience on a free and anonymous basis. Those who took part in the additional survey were asked to give feedback about each type of exercise (psychoeducation videos, content questions, scenario questions, video/photo questions). As a result of the feedback, refinements on the question instructions were made to improve the training experience. For example, in the reflection questions, some participants reported it was difficult to come up with emotion episodes from their own lives. Thus, we added instructions that better defined what constituted an emotion episode and gave examples they can draw on.
Table 4. Structure of pilot tested Training Content.
| Session # | Parts in session | Topic of Session |
|---|
| Session 1 | Video | Introduction to Emotions |
| Questions with two answer options | |
| Scenario exercises | |
| Reflection exercises | |
| Recognition exercises with Pictures | |
| Session 2 | Video | Emotion Components |
| Questions with two answer options | |
| Scenario exercises | |
| Reflection exercises | |
| Recognition exercises with Pictures | |
| Session 3 | Video | Emotion Regulation |
| Questions with two answer options | |
| Scenario exercises | |
| Reflection exercises | |
| Recognition exercises with Pictures | |
| Session 4 | Video | Twelve emotions – Part 1 |
| Questions with two answer options | |
| Scenario exercises | |
| Reflection exercises | |
| Recognition exercises with Pictures | |
| Session 5 | Video | Twelve emotions – Part 2 |
| Questions with two answer options | |
| Scenario exercises | |
| Reflection exercises | |
| Recognition exercises with Pictures | |
Results
Design: Integrating emotion theory, expertise, and gamification theory
After reviewing theoretical models and drawing on the expertise of authors and collaborators, we designed four types of exercises: 1) watching psychoeducation videos and answering content related questions, 2) recognizing emotions from photo and video expressions of actors, 3) identifying emotions and features of emotions from reading vignettes, and 4) reflecting on one’s own emotion experiences to apply the skills to daily life.
To increase engagement in the training, we drew on gamification mechanics in the design stage. The gamification of the learning approach uses elements from gaming to motivate and engage users in learning environments (Kapp, 2012). We identified several gaming elements that would be realistically implementable given our programmers time and resources. These included: progress mechanics (badges, points), immediate feedback, opportunities for mastery and leveling up, and scaffolded learning with challenges of increasing difficulty (Zichermann & Cunningham, 2011).
When building the training, we took into account the feedback from focus group participants regarding motivations, hooks, barriers, and usability. Thus, after building the conceptual model, we designed a training that includes a set of 10 gamified ‘sessions’ nested within four ‘challenges’ that each start with an animated psychoeducation video (Table 5). The four challenges covered different topics: introduction to emotions, parts of an emotion, regulating emotions, recognizing specific emotions and a wrap-up session which included blended emotions. The user navigates through each Challenge with the difficulty of questions increasing both within each session and across the 10 sessions. Mastery of the training is communicated through badges and the unlocking of new topics. Users first watch a psychoeducation video about the general topic. They then complete exercises about that topic. Users then identify emotions from scenarios, photo expressions and video expressions. Finally, they apply what they learn to their own life by reflecting on recent emotional experiences and answering questions about it. In this sense, the training educates users on the theoretical model, other awareness (identifying emotions in others) and self-awareness (reflecting on one’s own emotional experiences).
Table 5. Structure of Training Content.
| Name of Challenge | Session | Name of Session |
|---|
| Challenge 1 | What is an Emotion? | 1 | What is an emotion? |
| | 2 | Parts of an emotion. |
| Challenge 2 | Dealing with Emotions | 3 | Dealing with emotions Part 1 |
| | 4 | Dealing with emotions Part 2 |
| Challenge 3 | How to Recognise Emotions | 5 | Anger, Surprise and Anxiety |
| | 6 | Joy, Sadness and Fear |
| | 7 | Pride, Shame and Disappointment |
| | 8 | Pleasure, Disgust and Contentment |
| Challenge 4 | Emotions Wrap-up | 9 | Blended emotions |
| | | 10 | Emotions wrap-up |
Themes
The researcher (first author) who coded the focus group notes first identified larger themes, then specific comments within the themes, and the number of times each comment topic was brought up. Themes were derived by data and not identified in advance. No special software was used for this. During the content development and pilot testing stages, several themes were identified through participant feedback.
This feedback was incorporated incrementally at each step. Results below are organized by themes identified across focus group and Youth Advisory Board meetings, with quotes provided as specific examples. Feedback was only reported below when a group consensus was apparent in at least one focus group or meeting. For each quote, we identify in which of the focus groups (FG1-FG6) or YAB meetings (YAB1- YAB3) participants took part. Table 6 lists more specifics on how feedback was implemented or why it was not implemented.
Table 6. Implementation of Feedback.
| Theme | Feedback | Implementation |
|---|
| Personalization | Relatable content | Scenarios included with relatable situations |
| Including avatar that can be
personalized | Not implemented because of programming
complexity |
| Changing features of the app | Notifications and app color are adjustable |
| Choosing your own learning pace | Training divided into sessions that users do at their
own pace |
| Graphics and audiovisual elements | Professional quality audio | Professional voiceover actors hired for videos |
| Engaging and playful colors and tone | Videos were professionally animated |
| Fast stimuli | Videos were limited to a few minutes at most |
| Should not be too childish or
academic | Youth were involved in building and testing content
to avoid this |
| Interactivity | Personality type questionnaire | Not implemented because it is out of scope of the
training |
| Interactive story-telling | Not implemented because of programming
complexity |
| Having different levels of mastery | Training was divided into separate 'sessions' |
| Unlocking features | Sessions in training are 'locked’ until earlier
sessions are done |
| Too much gamification could be
distracting | Implemented in training |
| Rewards and badges for continued
usage | Implemented in larger app |
| Interest in tracking emotions | Implemented in larger app |
| Usability | Easy to navigate | Ease of use was tested before and after app
implementation |
| Script language may be too difficult
for 16 year olds | Scripts were edited and scientific terms were
defined and explained |
| Users expect to use app for 10–20
minutes each time | Sessions were designed to take less than 20
minutes |
| Privacy | Password can be required to enter
app | Passcode required in each log in for app |
| Any voice recordings should be
anonymous | Plan to pseudonymize voice recordings in larger
app |
| Concern that information is
confidential | Strict data security measures adopted |
| Legitimacy | Users should know how training will
help them | Explained in psychoeducation videos and
marketing materials |
| | Important to know if content is
approved by institutions | Scientifically grounded training developed by
researchers |
Personalization. The relatability of content appeared to be an important consideration for young audiences. When reviewing training content, focus group and pilot study participants added that psychoeducation videos would be interesting to young audiences if there were ample examples from daily life and that these examples be specific, realistic, and relatable.
For example, when asked what they would look for in a mental health app, one popular suggestion was to use scenarios of real-life situations that youth could understand and relate to in their phase of life, such as about love and romantic relationships. [FG1]
When discussing aspects of apps that would make them attractive, participants often mentioned personalization of app features. Personalization came up across a number of topics. For example, participants in one focus group brought up the idea of being able to set one’s own goals and one’s own pace, such as is done in exercise apps. Regarding notifications, one participant said:
“Notifications and reminders are ok, but make sure users can set their own desired frequency and what notifications [they receive].” [YAB2]
Participants also felt that different types of users may have different interests. For example, those of age 16 would have different interests than those aged 21, and that an app should have different features depending their own struggles and problems. Personalization of graphics came up across multiple focus groups, with ideas to be able to personalize app colors and visual representations (avatars) in an app. In one YAB meeting, participants commented that one way to personalize an app would be to include their name in notifications or when entering the app.
Graphics and audiovisual elements. Across a number of topics, the importance of certain graphic features was raised. At two focus groups in the early stages of the project, we asked participants what apps they currently use and what they like about the apps. Both groups mentioned the colors and visualizations in the apps they use are graphically pleasing, professional looking, and ‘not boring’ [FG1]. When asked what would make an app attractive, participants mentioned that graphics should be modern, with engaging colors, a fun layout and playful tone.
One participant summed up, “Graphics and fast stimuli are everything.” [FG3]
To gauge specific preferences for audiovisual elements, we showed participants seven different psychoeducation videos we pulled from the internet and asked them what they liked and disliked about the videos. Participants showed a preference for short videos, nice animations, simplicity, professional voiceovers, and appealing designs. There was a consensus in this age group that there shouldn’t be a ‘childish’ tone in educational videos. They also disliked videos that were too academic and that were too long.
Interactivity. In general, participants enjoyed interactive features of apps and training programs. This was brought up both spontaneously by participants and also when specifically asked about gamification features. When discussing features of apps they currently use, participants mentioned the ability to unlock parts of content and a progression of easy to difficult questions (such as those in language learning apps). Interestingly, there is a limitation to how gamified a self-improvement app should be, according to participants. Given that motivation to engage with the content is driven by a desire to gain knowledge and improve health, making the app completely gamified can be both unnecessary and detract from their aim.
As one participant put it, “Don’t make it a real game. If we want to play a game, we will download a game.” [YAB1, paraphrased].
Interactivity also came up in discussions regarding videos and usage reinforcement. When brainstorming ideas for video styles, one group discussed the idea of interactive stories where the user makes decisions throughout the story. Participants also believed rewards and badges would help encourage continuous usage, as it allows users to see their progress and reminds them of what they have accomplished in the app. Youth also liked the idea of getting insight and feedback from a training, akin to taking a personality test.
Usability. Across groups, usability was commonly reported as a desired characteristic of apps. Participants liked apps that were easily accessible, easy to use, and progressed quickly. For example, the complaint about a popular language app was that “sometimes it’s too slow.” [FG1] Another participant mentioned, “[i]t must be able to go quickly.” [FG2]
Speed was also mentioned in reviews of psychoeducation videos. When asked how long they would engage in a mental health app training per day, participants indicated an average of 10–20 minutes. In terms of usage, participants reported using apps at night (before going to bed) or when in waiting periods (e.g., on public transport).
Privacy. Another theme brought up by participants was consideration of data privacy. When asked what would make a mental health app attractive to them, the topic of privacy was brought up (e.g., that the app should be password protected). We also specifically asked about potential app features such as voice recording or sensor tracking and whether they would give permission to store such data if it was completely secure. There were mixed reactions; while some participants didn’t mind, others expressed distrust of privacy matters in general.
“The app creators can always say that the data are anonymized, but actually you simply don’t know what they really do with the data.” [FG4]
Another participant expressed security concerns from external sources, stating that, “you never know that the app gets hacked and that other companies can steal and use the information.” [FG4]
Legitimacy. When it comes to mental health apps, a sense of legitimacy mattered to participants of this age group. When asked what information they would like to know before using a mental health training, participants mentioned an approval by official organizations, institutions or universities. Two groups also mentioned wanting to know how exactly the training would help them.
Pilot testing: feedback on training
Pilot study participants completed an early version of the training on Qualtrics and provided qualitative feedback on each type of training exercise (psychoeducation videos, content knowledge questions, emotion recognition exercises, and self-reflection questions). Specific feedback allowed us to make changes on our particular training materials. For psychoeducation videos, participants liked clarity and simplicity in explanations and enjoyed colorful fun animations. A few reported disliking a slow pace, childish animation, and audio quality. For knowledge questions, participants liked that “it made you think about [what you learned]” and clear, unambiguous questions but disliked that they didn’t receive results on their performance (our low-fidelity prototype did not have feedback yet). For emotion recognition exercises, participants liked seeing diversity in faces in the photos, but reported that they needed clarification on certain emotions and disliked when multiple answers seemed possible. For self-reflection questions, participants generally liked the opportunity to reflect on their emotions with one participant saying, “The questions made you reflect on yourself and stimulated more self-insight.” However, some reported difficulties in coming up with emotion-evoking events from their lives or difficulties describing how they felt.
Discussion
Principle findings
The aim of this project was to build a mental health training that was both supported by science and appealing to young users. To address the need for a scientifically grounded training, we based the content on the Component Process Model (Scherer, 2009). We incorporated content and feedback given by youth to our theoretically guided design to ensure the training would be engaging to our 16–22 target user-base. We began with a conceptual model that integrated emotion theory and gamification concepts. We designed 10 sessions of content, nested into ‘Challenges’ that each began with an engaging, but short, psychoeducation video which was followed by emotion understanding, recognition, and identification exercises and ended in self-reflection questions. Sessions ‘unlocked’ and increased in difficulty. Youth in focus groups helped create some of the scenarios that were used in exercises so that they would be relatable and interesting to our age group.
In focus groups and Youth Advisory Board meetings, we obtained feedback regarding preferences in mental health trainings and apps. The most common themes that arose were personalization (being able to adjust features according to personal preference), relatability (content appropriate to age group), importance of graphics and audiovisual elements (colorful and fast), interactivity (rewards and feedback), usability (easy to use and accessible), privacy, and legitimacy (backed by institutions). We adjusted content iteratively throughout the project. In the pilot study where participants completed the training in a web-based space, we found that participants preferred clarity, unambiguous questions and answer choices, professional audiovisual quality, feedback on their performance, and the opportunity to self-reflect.
Reflecting on the process of developing the training, we found that working with a theoretical model (CPM) helped guide the structure of the training. Also, it was an easy and intuitive model from which to build an emotion-skills training, as it helped youth understand and dissect different parts of the emotion process. Because we developed the training with the participation and feedback of youth, we found it was possible to build a training that is both theoretically guided and appealing to users.
Considerations when developing a mental health training
The process of developing a mental health training for youth came with some lessons and challenges that can hopefully help guide future projects. First, despite the plethora of feedback in the very early stages regarding what young audiences want in a mental health app, there were clear limitations in the scope, timing and resources that prevented us in building a training that incorporated them all. Balancing the ideals and the realities will of course be specific to each project. It was nonetheless interesting and helpful to learn about preferences of our target audience at the very early stages of the process and we recommend this for future researchers.
Our age group included 16–22-year-old youth, who are at a particularly transitional life stage. This was reflected in their feedback, which in a few categories seemed paradoxical. Participants appreciated scientific and educational content but didn’t want it to be too academic. They wanted scenarios to be relatable but not too childish and preferred mental health apps that were interactive but not completely a game.
Lastly, we found it very useful to work in an iterative process, collecting and incorporating feedback at various stages of the process. This allowed us to make changes in early stages when it was still flexible and easy. Once content was programmed into the app, it had already been reviewed by focus groups and pilot study participants. This was useful because changes in the app itself required more time and resources. When the app was programmed, we checked for specific pain points and usability issues.
This training is included in the MyMoodCoach app that is being scientifically tested in a larger Randomized Controlled trial (RCT) as part of the ECoWeB Project. In future studies, the trial will test the effectiveness of the training when combined with other emotion competence trainings in a personalized mobile app approach. The stand-alone version of the training is also being tested in our lab.
Data availability
Underlying data
OSF: Development of a theoretically and empirically grounded emotion knowledge training for the young. 10.17605/OSF.IO/QSYGW (Hovasapian et al. (2022))
This project contains the following underlying data:
- Focus groups 1–3.docx (Notes from the first three focus groups.)
- Focus groups 4–6.docx (Notes from the next three focus groups.)
- YAB 1 _Sep 25 2018.docx (Youth Advisory Board meeting notes.)
- YAB 2 _April 19 2019.docx (Youth Advisory Board meeting notes.)
- YAB 3 _February 2020.docx (Youth Advisory Board meeting notes.)
- Pilot data.xlsx (Responses from optional feedback survey on training content within pilot study)
Extended data
This project contains the following extended data:
- Pilot TestingFeedback Survey.docx (File with questions from optional feedback survey in pilot study)
Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).
Ethics and consent
Ethical Approval was granted by the Ethics Committee of the Faculty of Psychology and Educational Sciences at Ghent University (2018/38/Arpine Hovasapian).
Informed written consent was obtained from participants of the focus groups and YABs, which included the use of their anonymized data. Passive parental consent was used for participants under 18 years in addition to their own informed consent.
Due to the completely voluntary basis and anonymous data collection of the optional survey within the pilot study, no informed consent was needed for this data collection, analysis and usage.
