Introduction
The increasing ageing of the global population, coupled with the rising prevalence of chronic conditions, poses significant challenges for healthcare systems worldwide. One response to these challenges has been the development and implementation of information and communication technologies (ICT), aimed at supporting active and healthy ageing, improving the quality of life, enabling remote monitoring and allowing older adults to live independently in their own homes for a longer period (Mostaghel, 2016). While many technologies already exist to support Ambient Assisted Living (AAL), such as systems for grocery shopping, communication with family and friends, robotic assistance systems, medication reminders, and wearable devices for fall detection (Hallewell Haslwanter & Fitzpatrick, 2017), these solutions often operate in silos and lack interoperability, resulting in fragmented and isolated social and healthcare services (Bygstad et al., 2015).
To successfully implement and integrate technologies, it is essential to understand the specific contexts of use, as well as the needs, tasks and goals of end-users and stakeholders. Participatory approaches, such as action research, have been developed to involve stakeholders as co-researchers (e.g. users, and professionals) in a cyclical process of planning, action, and reflection, thereby facilitating the introduction of change within communities (Oberschmidt et al., 2022). Implementing new technology in complex domains like social and healthcare requires careful adaptation to local contexts and to the characteristics and profiles of people within those contexts. Therefore, a close collaboration with the relevant stakeholders is essential to ensure successful integration and user acceptance.
Despite the widespread use of action research and user-centred design (UCD) in technology development, most studies focus on single technologies and often overlook the reality that users may simultaneously interact with multiple technologies and services in their daily lives. This gap highlights the need for testing integrated solutions in real-life settings (Lettieri et al., 2015) and underscores the importance of pre-validation to address potential challenges before large-scale deployment.
Furthermore, and while many projects implement a co-creation process, this co-creation is sometimes not applied to all phases in a project. Oftentimes stakeholders are consulted at the needs assessment and requirements elicitation phase, but then they are only involved again at a much more advanced stage of the technology development process. This means there is often a gap in involving the end-users in providing feedback to mock-ups and to partially functioning prototypes, which is a stage that is especially relevant to involve them in and consult them (for usability, UX, UI, etc.). When the technology validation with end-users is only conducted in the piloting stage, chances are that the solution will not have been properly designed to their needs, and increased efforts will be necessary to adjust it properly. We are considering socio-technical systems, where it is essential that the solutions are aligned with the existing care pathways, and with the underlying human dynamics, so they can seamlessly integrate into everyday life and not pose a burden. For this, pre-validation is essential. However, pre-validation with actual end-users, instead of healthy volunteers or even research team members, is still not always the norm.
In the Pharaon European project, the objective of the pre-validation was twofold: to assess the willingness of users to adopt the different technologies and identify any technical and usability issues at an early stage, before the commencement of the year-long pilots; and secondly, to test the feasibility of applying a standardised pre-validation methodology across six different sites. It was conducted in six pilot sites across five countries (Spain: Andalusia & Murcia; Italy; the Netherlands; Portugal; Slovenia) over a six-month period, involving a total of 191 test scenarios and coordinated by Gérontopôle Nouvelle-Aquitaine (France). These scenarios were based on specific challenges and usage cases defined earlier in the Pharaon project, ensuring that each pilot could effectively test the technologies with future users in conditions that closely resembled real-life settings (Mooses et al., 2022).
Overview of the pilot sites
6 Large-Scale Pilot sites and 10 targeted challenges
The pre-validation took place from 2021 to 2022, with a total of six months, in the six Large-Scale Pilot sites in five countries: Andalusia (Spain), Italy, Murcia (Spain), the Netherlands, Portugal and Slovenia. Since every pilot site used a different set of integrated technologies and services, many different use scenarios needed to be defined. They were, however, all tested in the same manner. A total of 191 tested scenarios were developed during the pre-validation period. They were based on challenges (CH) and usage scenarios defined at an early stage in the Pharaon project and represented an action that users had to manage during the test sessions organised. Indeed, user representatives identified ten specific challenges that each pilot planned to address during a previous work package of the project about Users requirements and Pilot requirements (Moose et al., 2022), (Tunc et al., 2023).
CH1 - The behaviour and the approach of older adults to friendly technological devices
CH2 - Health status definition and its progress over time
CH3 - Non-Intrusive Monitoring and Alarm Triggering
CH4 - Promoting social cohesion
CH5 - Defining specific personalised care plan based on user needs
CH6 - Reducing isolation and loneliness, enhancing autonomy through connectivity and digital tools
CH7 - Promoting accessibility and the provision of proximity services with IT platforms
CH8 - Promoting capacity building and awareness on green economy, citizenship and cultural traditions
CH9 - Indoor Environmental Quality
CH10 - Support to caregivers towards more efficient and personalised care services
An overview of the specific challenges identified and addressed by each pilot is depicted in Figure 1.
Figure 1. Overview of pilot sites challenges in the Pharaon project (This figure has been reproduced with permission from the Pharaon European project-H2020- Grant 857188.).
Through co-design sessions with users and regional stakeholders, each pilot site identified the specific needs, preferences and goals of the target group related to the challenges and based on these developed use case scenarios (extended data 1: Overview of each pilot goals, target group, technologies). The ecosystem of each pilot involved actors from the whole value chain, which ensures the proper development of each proposed scenario. At a later stage in the project, additional technology partners were included in each pilot via an Open Call to extend the socio-technical ecosystem and increase the impact and benefit of the pilots.
Example of the Italian pilot: goals, target group, use cases scenarios and technologies
Goals: The Italian pilot aimed to propose personalised integrated care for frail older adults. It aimed to promote communication, socialisation and active ageing among older adults and their formal and informal caregivers and to increase social inclusion. Furthermore, health management and monitoring were supported by enabling users to monitor their physical activity and the environmental status at home via technologies. Physical and cognitive stimulation was supported through technology which is also monitored over time.
Main target group: older adults, informal caregivers, professional caregivers
Technologies included in the pilot: TV box, tablet app and smartphone app that enable social cohesion (relation with relatives and caregiver), telepresence robot that allows older adults to communicate with medical staff, smartwatch to monitor older adults’ physical activity
Additional technology via a European Open Call: social network platform and digital application coupled to a dashboard for professionals to allow cognitive and memory stimulation for older adults. They could also stay connected with caregivers, family, health professionals and friends.
Use case scenarios:
Ethical considerations
A general ethics framework was developed within the project, outlining the requirements related to ethical standards and compliance with the General Data Protection Regulation (GDPR).
This pre-validation study was conducted in accordance with the principles outlined in the Declaration of Helsinki. Ethical approval was obtained from the local ethics committees or institutional review boards (IRBs) at each pilot site prior to the commencement of the pre-validation study.
All participants provided written informed consent (extended data 2: Example of one informed consent, Italian pilot) before participating in the study. The consent process was approved by the respective ethics committees at each site. Participants were informed about the study's objectives, procedures, potential risks, and their rights, including the right to withdraw at any time without any consequences.
To protect participant confidentiality, everyone was assigned a unique identification number known only to the corresponding pilot site, ensuring pseudonymisation of all collected data. Data were stored securely at each pilot site in accordance with national data protection regulations and the European Union's General Data Protection Regulation (GDPR). Only pseudonymised and aggregated data were shared with and analysed by Gérontopôle Nouvelle-Aquitaine.
A general ethics framework was developed within the Pharaon project, outlining the requirements related to ethical standards and GDPR compliance. Each pilot site ensured adherence to both EU GDPR and any additional national regulations. This framework guided all research activities to uphold ethical guidelines and to protect the rights and well-being of all participants throughout the study.
Methods
The ethics committees from each pilot site
Ethical approval was obtained from the local ethics committees or institutional review boards (IRBs) at each pilot site prior to the commencement of the pre-validation study. The details are as follows:
• Andalusia, Spain: Approved by the Ethics Committee of the University of Jaén, approval number [FEB.21/1.PRY], on March 2nd, 2021.
• Italy (Apulia): Approved by the Ethical Committee IRCCS Istituto Tumori “Giovanni Paolo II”, approval number [89/CE], on June 27th, 2021.
• Italy (Tuscany): Approved by the Ethics Committees of Tuscany Southeast, approval number [18626], on November 16th, 2020.
• Murcia, Spain: Approved by the Ethical Committee for Drug Research of the Virgen de la Arrixaca University Clinical Hospital, approval number [NE-2020-10-HCUVA], on January 27th, 2021.
• The Netherlands: Approved by the Ethics Committee Computer & Information Sciences from the University of Twente, approval number [RP 20201-194], on August 25th, 2021.
• Portugal (Amadora): Approved by the Ethics Committee for Health – SCMA, approval number [001A/CES/2021], on June 2nd, 2021.
• Portugal (Coimbra): Approved by the Ethics Committee for Health of the Regional Health Administration of the Centre, approval number [103/2020], on January 29th, 2021.
• Slovenia: Approved by the Slovenian Medical Ethics Committee, approval number [0120-306/2021/4], on August 3rd, 2021.
3 phases to pre-validate technologies with users
The pre-validation sessions were designed to evaluate both the willingness of end-users to adopt different technologies and to identify technical and usability issues. The methodology was harmonised across six pilot sites while accommodating site-specific needs.
The pre-validation was divided into 3 distinct phases, in which 18 technologies were assessed with a total of overall 222 users in the six pilot sites:
Phase 1 (September to October 2021): Individual technologies were tested to address specific challenges and use case scenarios.
Phase 2 (November 2021 and December 2021: Integrated technologies were assessed within personalised service platforms.
Phase 2B (November to December 2022): Technologies sourced through the Pharaon Open Call were tested to address unmet challenges identified during earlier phases.
The timeline for each phase was tailored to the specific circumstances of the individual pilot sites. In Phase 1, the schedule depended on the readiness of the technologies to be tested. In Phase 2, it was contingent upon the integration of these technologies into the respective pilot platforms. For Phase 2B, the implementation was initiated one year after the selection of new technologies through the Pharaon Open Call, addressing unmet needs that the initial partners could not fulfil from the project’s start.
To ensure consistency, Gérontopôle Nouvelle-Aquitaine, the coordinator of the pre-validation, established a standardised timeline for each phase. This timeline included three weeks for test sessions in each pilot site, followed by two weeks for the collection and synthesis of results, including drafting a comprehensive report for pilot site teams and technology providers. Finally, a three-week period was allocated for technical providers to address bugs and implement improvements. The overall timeline of the separate phases is depicted in Table 1.
Table 1. Overall timeline of the separate phases of the pre-validation.
| Year | 2021 | 2022 |
|---|
| Month | Sept | Oct | Nov | Dec | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sept | Oct | Nov | Dec |
| Pre-validation phases | Phase 1 | Phase 2 | | | | | | | | | | | Phase 2B |
Organisation of the pre-validation sessions with end-users
A general protocol was provided by the coordinator of the task to describe to the pilots the pre-validation task. It was containing definition and objectives of the pre-validation, the timeline, the users involved in each pilot, the methodology to proceed, the ethics requirements, the data management and each pilot specificities. The pre-validation process required several preparatory actions to ensure consistency and compliance across all pilot sites (Figure 2).
Figure 2. The different steps of the pre-validation preparation (This figure has been reproduced with permission from the Pharaon European project-H2020- Grant 857188.).
The Pharaon pre-validation methodology was designed to be conducted in 6 different pilots with different goals and specificities. Therefore, the idea was to find standardised tools for all pilots that were aligned with the pre-validation goals. Several parameters had to be considered as:
Pilots did not all have the same target group (job, disease or not, age, living in an institution or not).
The way to proceed with older adults was not the same as for a health professional. For example, the Murcia pilot worked with older adults and healthcare professionals, while the Dutch pilot collaborated with older community-dwelling adults and volunteers.
The ethics approval did not take the same time in all pilots as the organisations to be approached were not the same. This had an impact on the timeline.
An instruction booklet was delivered to each pilot with the requirement for the technical providers, the general pre-requisites, how to use Gitlab platform, the organisation of a test session and how to proceed with the data to transmit.
The three pre-validation phases were organised according to the same model (Table 2): three weeks to deliver the test session with users, two weeks to collect, analyse, compile and share the results with pilot sites and technical providers and three weeks to improve the solutions.
Table 2. Organisation of each of the 3 pre-validation phases.
| Test sessions | Collection of results and report by Gérontopôle
Nouvelle Aquitaine | Improvements and fixations of
technical bugs |
| Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Week 7 | Week 8 |
Methods for data collection and analysis
The methods for data collection included issue-tracking tools (reported also in Gitlab), the After-Scenario questionnaire (ASQ) and the System Usability Scale (SUS). Each pilot site managed their test session with their specificities (see section “Organisation of the pre-validation sessions with end-users”). In each pilot site, an evaluation manager was assigned to deliver the session with users and to collect the results.
Tools
Issue tracking using Gitlab
Gitlab is an Open Source code repository2 and collaborative software development platform for large software development projects. This platform was used by all technical providers concerning the development of the technologies and during the pre-validation, to report bugs and their fixation. Gitlab was used by the pilot team to report the technical issues identified during the test sessions in each phase. Different labels or milestones were created to improve the communication between pilot site teams and technical providers. All pilot sites created a user account during the preparation of the deployment of the pre-validation.
After-Scenario Questionnaire (ASQ)
Developed by J.R Lewis in 1991 (Lewis, 1991), this survey is widely used as it has simple statements on a 7-point Likert scale (Likert, 1932) used to assess how difficult a task is perceived during a usability test (Extended data 3: After Scenario Questionnaire).
A result above or higher than 5 was considered as good enough concerning the difficulty to use, even if some issues could still be there for the user. A comment section was added to the questions to collect some qualitative results with a description of the user experience. Most users were accompanied by a facilitator from the pilot site team to fill in the questionnaires and the comment section.
System Usability Scale (SUS)
The SUS (Extended data 4: System Usability Scale) is a well-known method developed by John Brooke in 1995 (Brooke, 1995), to have a general view of subjective assessments of usability. It figures in ISO 9241 standard and consists of 10 items with a Likert scale. The result is a score between 0 and 100. This score is obtained with the following equation:
SUS = 2.5 (20+∑ (SUS01, SUS03, SUS05, SUS07, SUS09) - ∑ (SUS02, SUS04, SUS06, SUS08, SUS10)
Within the Pharaon project, we defined sufficient usability as an average score higher or equal to 68 (Brooke, 1995). This would mean users were satisfied with her/his experience and accepted the solution. A comment section was added after each answer to allow users to provide qualitative feedback to the technical providers.
Data analysis
Next to the usability issues that were directly reported in Gitlab by the pilot team, each pilot sent the pseudonymised results of the questionnaires to the pre-validation coordinator (Gérontopôle Nouvelle-Aquitaine).
After collecting all results for each pilot site and technical solution, those results were analysed and averages calculated for each technology and platform’s ASQ and SUS in each pilot site to obtain quantitative results.
All results were then synthesised to have an overview of all quantitative and qualitative results. Moreover, a file dedicated to each technology was set up. This file containing all comments from the users enabled technical providers to outline the improvements they could plan to solve the identified issues.
Test session procedure
Each session (lasting 1.5 to 2 hours per user per technology) was conducted following a standardised protocol over a three-week period. This timeline allowed for adaptation to specific circumstances, such as the schedules of health professionals or the closure of establishments to the public due to COVID-19. The protocol included the following steps:
1. Pseudonymisation of the user, if not done before the session (pilot team)
2. Explanation of the session's purpose and procedures to the user (pilot team)
3. Presenting and explaining the informed consent form to the user, ensuring they fully understand its content and purpose, followed by obtaining their signature (user)
4. Providing instructions for the first tested scenario (scenario 1) and the corresponding solution (solution 1) (pilot team)
5. User testing of scenario 1 with solution 1 (user)
6. Documenting any technical malfunctions, if applicable (pilot team)
7. Completing the After-Scenario Questionnaire (ASQ) and providing comments, with support from the pilot team as needed (user supported by pilot team)
The sequence of steps 5 to 7 was repeated for all tested scenarios per technology. At the end of the testing for each technology, users completed the System Usability Scale (SUS) questionnaire.
Figure 3 to Figure 5 present examples of how the test sessions were held.
Figure 3. Photos from pre-validation phase 1, Slovenian Pilot (This figure has been reproduced with permission from the Pharaon European project-H2020- Grant 857188.).
Figure 4. Photos from pre-validation phase 1, Portuguese Pilot (This figure has been reproduced with permission from the Pharaon European project-H2020- Grant 857188.).
Figure 5. Photos from pre-validation Phase 1, Portuguese Pilot in Coimbra (This figure has been reproduced with permission from the Pharaon European project-H2020- Grant 857188.).
Tasks per technology tested in each pilot
Usability refers to the degree to which a system, product, or service enables specific users to achieve their goals effectively, efficiently, and satisfactorily within a defined context of use, as outlined in ISO 9241-210:2019. In line with this definition, each pilot site, in collaboration with the pre-validation coordinator and technological partners, was required to define tasks or tested scenarios that the user could perform with the technology being tested. Two examples of tested scenarios during phase 1 are presented below (Table 3), focusing on the pilot sites in Murcia and the Netherlands, which had different challenges and involved different user profiles. The number of scenarios and technologies tested varied according to the needs and objectives of each pilot site.
Table 3. The example of 2 pilots’ scenarios tested in phase 1 during the sessions with the users.
| Pilot site | Technologies | Tested scenarios |
|---|
| The Netherlands | Digital tool to advice about physical activity to increase older adults’ empowerment | • You want to know your active minutes of yesterday
• You want to see if you have reached your daily goal today
• You want to fix a new goal of daily steps (example: 2500 steps per day |
| Digital tool for advertising older adults about food | • You want to record in the application the last lunch you had to define your eating habits
• You want to consult and choose a realistic goal to change your eating habits
• You have run out of ideas for your next dinner. You want to find a new recipe on the App
• You want to listen to Mr XX’s testimony about sport lessons |
| Digital tool allowing older adults to relate to their transport platform and sign up for a trip or an exchange with other participants | • You want to check if there is a trip planned to the Space Expo Museum and sign up for it
• You go to your profile to check your next activities
• You realise that you are already busy on the day of the trip and decide to cancel your participation
• You realise that you have recently moved and that your address is not correct. You want to change it. |
| Murcia | Monitor and collect information from older adults’ environment and activities. Informal caregivers and health professionals are alerted if the information collected is abnormal | • You want to find out the time you went to bed on the specific date according to the App (older adult)
• You want to know the humidity in your home today (older adult)
• You want to check that Mrs X’s movements in her flat have been regular over the last 3 days (Informal caregiver/Health professional)
• You want to check that you receive an alert when the alarm configuration is fulfilled (informal caregiver/Health professional) |
| Older adults’ electricity consumption and alerts to caregivers in case of older adults non usual activities | • You want to know in real time electricity consumption (older adults).
• You want to know if the kitchen has been used for cooking (older adults).
• You want to know which electrical device consumes the most electricity in your home (older adult / informal caregiver).
• You want to see if you have received any alarm or events about changes in power consumption patterns of your assigned older adult (informal caregiver / health professional).
• You want to know the electricity consumption of MrX yesterday (healthcare professional).
• You wonder if Mrs X is spending more and more time in front of her television. You want to check this information (healthcare professional). |
| A robot for telemedicine with isolated patients | • Your appointment with Mrs X is in 15 minutes so you want to remote control a robot to her room to do the teleconsultation (healthcare professional).
• Then you want to begin the teleconsultation (healthcare professional). |
| A smart band to allow older adults to consult heart rate and steps in real time | • You want to consult your heart rate / the rate of Mr X (older adult / informal caregiver).
• You want to know how many steps you have made today / or Mr X (older adult / informal caregiver).
• You want to check the evolution of the patient's heart rate (healthcare professional).
• You want to know how many steps Mrs X has made yesterday (healthcare professional). |
Results
Overview of participants
Results were both quantitative, according to the questionnaires with a Likert scale, and qualitative, according to the comment section present after the questions asked to the user.
An overview of participants involved in the pre-validation is shown in Table 4, which presents a synthesis prepared by Gérontopôle Nouvelle-Aquitaine for this paper.
Table 4. Users involved in the pre-validation for each pilot.
| Participant profile | Andalusia | Italy | Murcia | The Netherlands | Portugal | Slovenia | Total |
|---|
| Older adults | 14 | 17 | 9 | 2 | 25 | 28 | 95 |
| Informal caregivers | 1 | 9 | 7 | | 9 | 11 | 37 |
| Formal caregivers | 13 | 16 | | | 23 | 12 | 64 |
| Health professionals | | | 19 | | | | 19 |
| Volunteers | | | | 3 | 4 | | 7 |
| Total | 28 | 42 | 35 | 5 | 61 | 51 | 222 |
Moreover, different profiles of participants were involved according to the challenge of the pilot site and different technologies being tested to address it. A minimum of 5 people per type was required for a user test. This number was chosen based on research indicating that 5 users can identify approximately 84% of usability issues, while 15 participants are ideal to capture 100% (Woolrych & Cockton, 2001). Given the number of pilot sites, the variety of technologies, and the range of user types involved, Gérontopôle Nouvelle-Aquitaine opted for the minimum threshold of 5 participants per type to balance feasibility with comprehensive feedback.
Quantitative results
A total of 109 bugs were highlighted during the three phases of pre-validation and most of them were solved in the following weeks.
In Phase 1: 6/18 technologies had all their functionalities working and were considered useful for participants. Only 1 technology did not meet users’ needs. For the other 12 technologies, results were mixed, and improvements were suggested by participants, facilitators, or the task coordinator, based on the comment sections in the questionnaires.
In Phase 2: 1 integrated technology and 1/8 pilot platform had all their functionalities working and were considered useful for participants. For other integrated technologies and pilot platforms, results were more mixed, and new improvements were suggested by participants, evaluation managers or the task coordinator.
In Phase 2B: 4/10 technologies had results indicating no difficulty and good user acceptance. No technology had results completely under the acceptance score even though one in Portugal did not achieve a good score in the SUS. This technology was not as mature as the other technologies and required significant adaptation of its features to meet the pilot site’ s needs.
For the technologies with mixed results, improvements were suggested by participants, based on the comment section in the questionnaires.
Qualitative results
According to the comment section integrated after each question of the After Scenario Questionnaire (ASQ), some improvements which were not technical but more ergonomic could be engaged by the technical providers.
Comments included:
A better visualisation of text and icons
A better understanding of text and icons
A better explanation of the interpretation of the data
Change the wording of the application to increase clarity
All comments were sent to the technical providers to allow them to make improvements before using the technology for 12 months during the Large-Scale Pilots.
Lessons learned of the pre-validation process
Lessons learned about the methodology appeared after the pre-validation task. Indeed, during the first phase, the users mentioned the complexity of the SUS items and expressed difficulties in understanding them, or the excessive length of the sessions due to the repetition of the ASQ after each tested scenario. There was also the importance of having facilitators with users according to the imbalance between users’ skills regarding the use of technologies (Lorusso et al., 2023).
Key People, Communication, and User Involvement
Caregivers played a pivotal role in several pilots, serving as the primary link between researchers and older adults. In the Slovenian pilot, they were essential in building trust and eliciting honest feedback, minimizing socially desirable answers. However, formal caregivers often faced challenges related to workload, while informal caregivers expressed mixed feelings about the usefulness of the technologies. Similarly, in the Italian pilot, facilitators with a positive attitude were critical for supporting older adults, particularly those with low digital skills, during the long deployment phase. In the Murcia pilot (Melero-Munoz et al., 2022), informal caregivers provided indispensable support but often felt overwhelmed by frequent alerts and additional responsibilities.
Effective communication was highlighted as a significant factor in the success of the pre-validation process. Flexible scheduling accommodated diverse preferences: formal caregivers required greater flexibility, informal caregivers preferred sessions in the afternoon or evening, and older adults favoured morning sessions. In the Portuguese pilot, structured communication tools, such as color-coded Excel sheets, were introduced to streamline coordination and enhance clarity in data exchange between pilot sites and technology partners. These tools proved particularly effective for consolidating qualitative and quantitative feedback into actionable insights for technical adjustments. Consolidated feedback, combining qualitative and quantitative data, was shared with technology providers to facilitate better understanding and alignment.
Co-creation efforts played a significant role in improving usability and engagement across pilots. For instance, involving older adults in creating promotional materials provided them with a sense of ownership, while collaborating on technology design ensured the solutions were more aligned with user needs and preferences.
Tools, Questionnaires, and Training
The evaluation tools used during the pre-validation sessions emerged as a common area for improvement, particularly regarding their repetitiveness and lack of clarity. Participants across multiple pilots expressed frustration with the ASQ questionnaires, which were seen as disengaging. Preliminary evaluations of tools were recommended to ensure they were understandable and engaging for all participants. Dynamic and interactive tools were suggested to reduce disruptions caused by low literacy levels and to maintain participant motivation. Standardized paper-based materials, combined with accessible support options such as phone numbers, were preferred by older adults over digital resources.
Training and digital literacy were equally critical. Tailored support ensured meaningful participation from diverse groups with varying levels of education and digital skills. Developing training content that promotes autonomy and usability enabled older adults to independently engage with the technologies. This included user-friendly, autonomy-focused materials designed to empower users while addressing proficiency gaps.
Technical Challenges and Adaptation
The pre-validation process highlighted several technical challenges that needed resolution to enhance the usability and reliability of the technologies. Adjustments to SOS alert configurations and refining functionalities to align with professional workflows were essential in the Murcia pilot. Connectivity issues, excessive alerts, and bugs disrupted operations in several pilots, leading to disengagement among older adults. Customization and adaptability proved vital in ensuring the relevance of solutions in diverse settings. For example, the Slovenian pilot demonstrated the importance of flexible scheduling and tailored support, while the Dutch pilot emphasized robust, user-centric system designs to meet varied user needs.
Autonomy and user confidence
Promoting user autonomy was a recurring theme across pilots. The Andalusian pilot aimed to ensure that older adults could engage with technology independently, without constant reliance on caregivers. Similarly, the Dutch pilot highlighted the value of printed materials and phone support options to provide users with confidence and security while navigating the technologies. By enhancing cognitive ergonomics and usability, pilots supported users in becoming more self-reliant.
Discussion
The pre-validation of the various technologies and their integration into the pilots’ platforms revealed several necessary and potential improvements, both ergonomic (such as user interface design, including colours and font sizes) and technical (such as feature development). Each pilot site was able to test the technologies with its future users over a short period, simulating real-world conditions as closely as possible. This approach helped to identify potential issues early on, enabling a more secure setup for the Large-Scale-Pilot and reducing the risk of user disengagement due to technological failures. Indeed, Large-Scale-Pilots were set up for 12 months and any technical malfunctions during this period could significantly diminish users' motivation to continue participating to better understand these reactions, a technostress questionnaire was employed across multiple pilot sites. This assessment highlighted the need for targeted interventions to build digital confidence among participants before the full deployment of the technologies.
These initial phases also helped to better define the target groups for each technology. For some older adults, the complexity of certain functionalities proved to be too challenging, indicating that these technologies may not be suitable for widespread deployment among this population. As a result, some pilot sites have been preparing training programmes for informal caregivers, who will then support the older adults in their use of the technology, providing a more accessible and effective learning environment.
The pre-validation phases also demonstrated that digital literacy levels among targeted users varied significantly. Many participants needed foundational training on basic tasks, such as using a tablet or mobile phone, before they could engage meaningfully with more advanced functionalities. This finding underscores the importance of implementing introductory training sessions as part of the deployment strategy.
Additionally, the setup of Phase 2B, which incorporated technologies from the Pharaon Open Call, confirmed that even market-ready solutions must undergo thorough user testing to ensure they meet the specific needs and expectations of the intended users. This step is crucial to achieving the project’s goals and ensuring the successful adoption and long-term use of the technologies.
Conclusion
This article outlines the pre-validation methodology implemented across six pilot sites in five countries, structured in three different phases, in the Pharaon European project. The harmonised approach allowed for a consistent evaluation of diverse technologies and platforms, while still accommodating the specific needs and contexts of each pilot site. The use of the standardised tools was crucial in gathering comparable data and providing actionable insights, aligned with the action research.
However, the pre-validation process also highlighted several challenges, including disparity between expected research outcomes and real-life scenarios. It became evident that while standardised methodologies are essential, flexibility and adaptation to local contexts are equally important to ensure meaningful results. In particular, standardised validations should be tailored to the target population (especially for older adults) and to the specific countries or regions involved. This would help ensure that the technologies used are both culturally and practically relevant to the users they are designed for, allowing for more accurate and impactful outcomes.
One of the key takeaways from this pre-validation experience is the critical role of facilitators in maintaining continuous engagement with users throughout the testing phases. Their involvement not only helped bridge the gap between the technical and practical aspects of technology use but also ensured that users felt supported and understood. This was particularly important given the diverse digital literacy levels and the varying levels of comfort with technology among older adults.
Moreover, the pre-validation confirmed that even market-ready technologies require rigorous testing with end-users to identify unforeseen issues and refine functionalities. This step is crucial before deploying these technologies in Large-Scale-Pilots to avoid potential user frustration and disengagement. The inclusion of feedback from all stakeholders (older adults, caregivers, and healthcare professionals) proved invaluable in making iterative improvements to the technologies and adapting them to real-world use.
In conclusion, the pre-validation process has been a crucial step in fine-tuning the technological solutions within the Pharaon project, offering valuable insights into the complexities of deploying these systems across different European contexts. The knowledge gained will be instrumental in shaping future Large-Scale-Pilots, ensuring that the solutions are more user-centric, flexible, and effective in supporting the health and well-being of older adults and their caregivers.
Ethics and consent
A general ethics framework was developed within the project, outlining the requirements related to ethical standards and compliance with the General Data Protection Regulation (GDPR).
This pre-validation study was conducted in accordance with the principles outlined in the Declaration of Helsinki. Ethical approval was obtained from the local ethics committees or institutional review boards (IRBs) at each pilot site prior to the commencement of the pre-validation study. The details are as follows:
• Andalusia, Spain: Approved by the Ethics Committee of the University of Jaén, approval number [FEB.21/1.PRY], on March 2nd, 2021.
• Italy (Apulia): Approved by the Ethical Committee IRCCS Istituto Tumori “Giovanni Paolo II”, approval number [89/CE], on June 27th, 2021.
• Italy (Tuscany): Approved by the Ethics Committees of Tuscany Southeast, approval number [18626], on November 16th, 2020.
• Murcia, Spain: Approved by the Ethical Committee for Drug Research of the Virgen de la Arrixaca University Clinical Hospital, approval number [NE-2020-10-HCUVA], on January 27th, 2021.
• The Netherlands: Approved by the Ethics Committee Computer & Information Sciences from the University of Twente, approval number [RP 20201-194], on August 25th, 2021.
• Portugal (Amadora): Approved by the Ethics Committee for Health – SCMA, approval number [001A/CES/2021], on June 2nd, 2021.
• Portugal (Coimbra): Approved by the Ethics Committee for Health of the Regional Health Administration of the Centre, approval number [103/2020], on January 29th, 2021.
• Slovenia: Approved by the Slovenian Medical Ethics Committee, approval number [0120-306/2021/4], on August 3rd, 2021.
All participants provided written informed consent (extended data 2: Example of one informed consent, Italian pilot) before participating in the study. The consent process was approved by the respective ethics committees at each site. Participants were informed about the study's objectives, procedures, potential risks, and their rights, including the right to withdraw at any time without any consequences.
To protect participant confidentiality, everyone was assigned a unique identification number known only to the corresponding pilot site, ensuring pseudonymisation of all collected data. Data were stored securely at each pilot site in accordance with national data protection regulations and the European Union's General Data Protection Regulation (GDPR). Only pseudonymised and aggregated data were shared with and analysed by Gérontopôle Nouvelle-Aquitaine.
A general ethics framework was developed within the Pharaon project, outlining the requirements related to ethical standards and GDPR compliance. Each pilot site ensured adherence to both EU GDPR and any additional national regulations. This framework guided all research activities to uphold ethical guidelines and to protect the rights and well-being of all participants throughout the study.
Data and software availability
As this article is focused on methodology and most of the results are qualitative, there is no data repository to share.
No data associated with this article.
Extended data
Dryad: Methodological material used for pre-validation study in H2020 Pharaon project; Doi: https://doi.org/10.5061/dryad.2280gb63h (Lesterpt (2025))
This project contains the following extended data:
Extended data 1: Overview of each pilot goals, target group, technologies and uses cases scenarios
Extended_data_2_Example_of_one_informed_consent__Italian_pilot_EN.pdf
Extended_data_3_After_Scenario_Questionnaire.pdf
Extended_data_4_System_Usability_Scale.pdf
README.pdf
Data is available under the terms of the Creative Commons Zero v1.0 Universal
Acknowledgements and funding
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 857188 (Pharaon Project).
The authors want to thank all participants in the pre-validation of the Pharaon project. A special thanks to the general pilot coordinators Lex van Velsen, Christiane Grünloh, Kira Oberschmidt in charge of the Pharaon action research task and Filippo Cavallo as the Pharaon project coordinator. All these people were partners in the Pharaon project and were aware of this article coming from the project.
Faculty Opinions recommendedReferences
- Bygstad B, Hanseth O, Truong Le D:
From IT silos to integrated solutions. A study in e-health complexity. ECIS 2015 Completed Research Papers. Paper 23, 2015. Publisher Full Text
- Brooke J:
SUS: a quick and dirty usability scale.
Usability Eval.
Ind. 1995; 189.
- Hallewell Haslwanter JD, Fitzpatrick G:
Issues in the development of AAL systems: what experts think. 2017; 201–208. Publisher Full Text
- Lesterpt I:
Methodological material used for pre-validation study in H2020 Pharaon project [Dataset]. Dryad. 2025. http://www.doi.org/10.5061/dryad.2280gb63h
- Lettieri E, Fumagalli LP, Radaelli G, et al.:
Empowering patients through eHealth: a case report of a pan-European project Health policy, reform, governance and law.
BMC Health Serv Res.
2015; 15: 309. PubMed Abstract
| Publisher Full Text
| Free Full Text
- Lewis JR:
Psychometric evaluation of an After-Scenario Questionnaire for computer usability studies: the ASQ.
SIGCHI Bull.
1991; 23(1): 78–81. Publisher Full Text
- Likert R:
A technique for the measurement of attitudes.
Arch Psychol (Chic).
1932; 140. Reference Source
- Lorusso L, Mosmondor M, Grguric A, et al.:
Design and evaluation of personalized services to foster active aging: the experience of technology Pre-validation in Italian Pilots.
Sensors (Basel).
2023; 23(2): 797. PubMed Abstract
| Publisher Full Text
| Free Full Text
- Melero-Muñoz FJ, Bueno-Delgado MV, Martínez-Carreras R, et al.:
Design and development of a heterogeneous active assisted living solution for monitoring and following up with chronic heart failure patients in Spain.
Sensors (Basel).
2022; 22(22): 8961. PubMed Abstract
| Publisher Full Text
| Free Full Text
- Mooses K, Camacho M, Cavallo F, et al.:
Involving older adults during COVID-19 restrictions in developing an ecosystem supporting active aging: overview of alternative elicitation methods and common requirements from Five European Countries.
Front Psychol.
2022; 13: 818706. PubMed Abstract
| Publisher Full Text
| Free Full Text
- Mostaghel R:
Innovation and technology for the elderly: systematic literature review.
J Bus Res.
2016; 69(11): 4896–4900. Publisher Full Text
- Oberschmidt K, Grünloh C, Nijboer F, et al.:
Best practices and lessons learned for action research in eHealth design and impleentation: literature review.
J Med Internet Res.
2022; 24(1): e31795. PubMed Abstract
| Publisher Full Text
| Free Full Text
- Tunc S, Nijboer F, Tinga AM, et al.:
” Hi, My Name is Robin” – Remotly co-designing an embodied conversational agent for empathy with older adults. 2023; 376–91. Publisher Full Text
- Woolrych A, Cockton G:
Why and when five test users aren’t enough. 2001. Reference Source
