Introduction
Background
CRANE is an internet and digital based healthcare Pre-Commercial Procurement (PCP) project, i.e. Horizon 2020 project (Horizon, 2020). The aim is to develop an integrated and flexible self-management system for treating chronic patients at home in rural areas, thus improving the quality of life of patients, and optimizing health and care resources. The goal has been to run an iterative development process, and to create a Value-Based solution. The iterative development approach requires that the actors test, evaluate and adjust their hypothesis, designs and solutions throughout a process from idea to implemented solution (Ries, 2011).
Value-Based Procurement (VBP) is increasingly becoming part of the innovation procurement processes (Boston Consulting Group, 2024). VBP must focus not only on the price of a particular product or service, but also on the overall value of the solution it can create, in terms of improved patient outcomes, reduced total cost of care, and benefits to stakeholders such as healthcare workers (Boston Consulting Group, 2024).
Learning and improvements from running the PCP process has been part of the CRANE project design. A total of five workshops with the partners and management team running the process are planned.
The basis for this study
CRANE is run by 7 partners with multidisciplinary backgrounds from 4 European countries and three rural regions- Region Västerbotten in Sweden, Extremadura in Spain, and Agder in Norway Consortium | CRANE. The project is divided into three phases as shown in Figure 1 below.
Figure 1. The CRANE Pre-Commercial Process (PCP).
Twenty providers shown in Figure 2 responded on request for tenders participating in the CRANE PCP process. Five of these providers were selected to develop a final design of their suggested solution, i.e. phase 1 of the CRANE PCP process. These five providers were: Combitech AB (dHDSI), Data for Good (Data for Care), Egde Consultants AS (Antigone), Televes S.A.U. (MyRuralHealth) and Wiring Technologies SL.
Figure 2. Tenders received from a list of providers on 20th of January 2023.
We used a framework for early Health Technology Assessment (eHTA) (Ørjasæter et al., 2025) to select the five providers continuing to the next phase of the CRANE PCP process. eHTA evaluates technologies still in development and can be defined as the early examination of a health intervention’s potential value in terms of medical, social, economic and ethical issues. Thus, eHTA is a suitable prioritization tool in the early development stages before a solution is final. Key Performance Indicators (KPIs) were developed based on the eHTA framework and used as selection criteria.
Aim of this study
This Lessons Learned study aims to help the regional management team with knowledge sharing, giving these actors insights into methods used, i.e. identifying positive- and negative factors, improving the process of developing project documents, inviting providers and developing design solutions in the CRANE PCP process.
Methods
Context
A qualitative approach gathering data has been used in this Lessons Learned study. Data was collected in a workshop run on the 15th of April 2023 with 15 participants (Figure 1) from the CRANE partners responsible for running the CRANE PCP process. The participants were all health care representatives.
The workshop was led by the corresponding author and data collected by referees from the partners. The corresponding author is a research partner of CRANE PCP and has interest in the interpretation of data.
The data was collected in two sessions, one concerning cooperation and teamwork (
Table 1) and one concerning the deliverables (
Table 3). The collected data was sent to the corresponding author, and thereafter structured and processed into main topics. Subsequent input and feedback were given by some of the workshop participants based on the processed data. Improvements were based on the positive and negative factors. These are shown in
Table 2 for factors concerning cooperation and teamwork and
Table 4 for factors concerning the deliverables.
Results
The workshop gave important insights into potential positive- and negative factors in the phase where the five providers were to develop a design solution. The participants were all asked to bear in mind the patient and healthcare worker perspectives giving inputs.
Improvements were also identified based on the positive and negative factors. The positive and negative factors, and improvements, are listed in the table shown at the end of the paper:
•
Table 1: Positive and negative factors connected to cooperation and teamwork among the partners. The inputs are structured in five main areas.
•
Table 2: Improvements connected to cooperation and teamwork in the start and the first phase of the CRANE PCP process. The improvements are based on the positive and negative input and discussions among the participants in the Workshop. The improvements are structured in four main areas.
•
Table 3: Quality of the deliverables from the partners and providers. The inputs are structured in six main areas.
•
Table 4: Improvements connected to the deliverables in the start and the first phase of the CRANE PCP process. The improvements are based on the positive and negative input and discussions among the participants in the Workshop. The improvements are structured in three main areas.
Table 1. Positive and negative factors connected to the cooperation and teamwork.
|
Cooperation and teamwork
|
|---|
| Positive factors | Negative factors |
|---|
The importance of meeting physically
• The kick-off meeting in Copenhagen and the meeting in Extremadura was extremely important • The impact of face-to-face meetings was so much higher than the online meetings • We work when we meet • Over timer we have built trust in the project team. Physical meetings have been important building this trust. We have found each other • Very productive during our physical meetings – good dialogue
Dedication and learning
• All partners have been very committed • Strong motivation has been shown from all partners • Cooperation is great and we have learned a lot • The learning of partners has become very good
Capability of change
• We have been able to adjust and learn from each other • We have been very agile developing and changing the scope • Sharing knowledge with each other • Great dynamic and free speak – sometimes ‘hot’ but seems to be okay • Worked differently – found our own way • We solved problems on the run and when needed
Competence
• Strong professionals • Multidisciplinary team – a lot of capabilities and competences in the project • The project has developed to be more technically oriented then original planned • We had to improve the technical competence into the project team, i.e. handled but a bit late • We have competencies that have been very aligned with EHDS
In general
• International project set-up is very good • Manage to use PCP as a tool! • Well-coordinated application phase • Although all last-minute changes and updates we manage to get a solid partnership that is strongly aligned and trust each other • The evaluation process and selection of the tenders was very well • Excellent support by knowledge partners also on admin support • We should be proud, a complexed project with late developed partnership. We got 21 bids, which is quite a success
| Change of focus
• Difficult to change the scope of the project, from more service to more technological scope • Unbalance of the project; change of focus • Frustrating for some of the stakeholders changing the focus/scope • Very late to find a tech expert. The tech expert entered the project just before the submission
Change of partners
• Changing partner from Norwegian Centre for e-health research to OUS and just 2 days before the submission • This late entrance also caused some delays, and challenges. Difficult to start up and have a common view
Partners
Work situation
• We were working separately but after meeting we were working more collectively and smoothly • Difficult with deliveries during holiday • The project team did not meet physically before June 2021 due to Covid • Some unbalance in the application, added to some difficulty since workplan and team was not addressed
Admin: structure, planning and communication
• Challenge to coordinate the partners • Poor structure in the meetings, especially in the beginning • Deadlines were tight and could have been planned better Poor structure and deadline • Unclear what to do, and by whom • Difficult to follow the emails and many meetings and some meetings are too long • Information coming from different sources and not one Project platform – project space • Time management is not the best and therefor the feedback on the work done is very difficult to give quality feedback • Didn’t manage to start working as soon as we would have needed • Lack of information flow in right time • Tight time plan in many WPs • Better budget for the tech team
In general
• Experienced some tension in the beginning • Takes time learning from each other • A very technical focus • More health professional recruited for the workshops should have been done • Hard to explain our project, especially the technical parts
|
Table 2. Improvements connected toed to the cooperation and teamwork.
|
Improvements
|
|---|
|
Cooperation and teamwork
|
|---|
1) Common level of understanding
• It’s important to develop and agree upon a vision early. Defining the vision should be the first step. Define the focus and vision from the beginning. Agreeing on the vision and focus early! • Establishment of a common level of understanding early of the vision, the concept, focus and project goals. Clear focus from the beginning
2) Physical meetings
• It’s important to have physical meetings early and frequently. Digital meetings are not enough. The impact of the face-to-face meeting was tremendous – social part important • Earlier kick off and first meetings, physical meetings are superior – This secures a joint point of departure and common view • We should have used more time in the beginning to learn to know the different partners • Better interaction during the application phases • Meet physically more often to align and keep developed trust
3) Planning and coordination
• More efforts to make a workplan consistent with the developed scope. We should have been better in planning and sticking to the plan! Better planning to meet deadlines! • How to follow the time plan is difficult with different cultures/personalities. Stick to agree plans • Improve the Project Handbook to handle changes and challenges in the project • Agree on the project handbook and follow it! • Clear roles and expectation descriptions • What the role of the different partners are, and what to deliver when is of crucial importance. We should have spent more time to update the workplan to make it clearer and more understandable • Revise the workplan frequently, since agile development may require updates to description • Better organize the work for the non-leading partners, so they know when to contribute, i.e. during an early workshop • The process of reviewing documents should have been more structured • Structuring how to cooperate should have been better from the beginning. The coordinator role is of crucial importance in a project like Crane, i.e. making process lean and efficient • More communication, both internal and external • Important to find a good date for the meetings – Plan for holiday periods. The holidays are sometimes different from the partners home countries were • Agree on and use a cooperation tool • More traditional governance structure – separate consortium meetings from other meetings • Less democracy in some ways. Not everyone needs to attend every meeting • It has been a challenge to coordinate work packages. An overall lack of EU project experience is partly the reason for this challenge. We should have been better in learn from each other, i.e. building on different experience of EU participation. Some of this problem was problems meeting physically due to Covid
4) Competence
|
Table 3. Positive and negative factors connected to the deliverables.
|
Deliverables
|
|---|
| Positive factors | Negative factors |
|---|
Commitment and interest
• Strong interest among industry. Many bids… • Managed to propose a call for tender that was unique and selling • We have addressed a field, that experienced actors feels that we need to solve • Users onboard • Challenge is fun and makes us improve!
An innovative project
The documents
• High quality of the deliverables • There has been put a lot of effort in creating the deliverables • Many pages but good result • We did deliver in the end but not in always in time • On the positive note the TD 1 & 2 seems to be understood by the contractors – most of them are happy • D6.1 is very good, but we did not spend enough time to go through it and reflect in the team • Content of deliverables was changed when needed • We managed to deliver required documents
| Change
• Some of the deliverables where not tailored to the new scope • Created some inconsistency due to change of scope • Might disrupt industry with our requirements
Needed skills and experience
• Lack of tech skills were a clear challenge. Lack of technical skills • Lack PCP experience, and Horizon 2020 procedure. • Lack of alignment between Insights User needs, challenges and requirements
Complexity
• Difficult and why and what the content • Ended mismatch between different requirements • Difficult to understand the core of Crane
Bidders
• 20 applications were too much • Some of the bidder missed the core of the project • Should have had a plan B handling so many bids • Maybe it would have been easier to state that the aim is to provide access health data or activity data to citizens • The pre-evaluation of submitted bids should have been better
The documents
• Sometimes difficult to understand and what should be included and why. We should have had an overview of the deliverables and why • Sometimes difficult to understand who is doing what within the documents, i.e., who is responsible for processing what content. We were a bit lost in the beginning • Some mismatches between documents • The TD 1 and TD 2 documents is very long and maybe too long – with an improved quality review process they would be more sharp • Made to many documents. High amount of deliverables • Lack of quality control and alignment between deliverables • Andre doc was not necessarily due changing the focus
The process of developing documents
• A lot of work after the review • Not enough time for feedback. No time for reviewing deliverables • Many meetings were consortium meetings, not WP-based • Hard to find a model to get input from the other WPs • Another governance model should have been used • Difficult to understand the rolls & responsibilities of the buyer regions • Difficult to know what and when to deliver when you are not task lead – especially when you need to gather internal resources that are short of time • Lack of vision influenced the deliverables in the beginning
|
Table 4. Improvements connected to the deliverables.
|
Improvements
|
|---|
|
Deliverables
|
|---|
5) Documents
• Some of the documents should have been a joint deliverable, evaluation framework + requirements + selection criteria’s • Better overview of the required documents (internally) • Better alignment between different deliverables • The commission should have given more clear requirements
6) Planning
• Better planning to meet deadlines • We must stick to agree plans • The digital meetings should have been structured and implemented • All meetings should have a specific agenda, not just for general discussions. But there should be a mix! • A standard process on how a PCP process works should have been made and shared internally
7) The process of developing documents
• Meet physically more often to align results of deliveries • Secure quality and review reading • Present deliverables to the project team to ensure alignment and responsibilities • Better distribution/organization of deliverables in the WPs, and the timing • Ensure common level of understanding among the participants on what to deliver when, and who is responsible for what part of the document • For the evaluation and the development of deliverables it could have been good to have more technical experts involved • We should consider in using an external feedback team for ensuring quality control. Quality management from start – in cooperation! • Better guidelines from the Commission • Balance good enough vs. always perfect (perfection vs. pragmatic) • Focus more on quality management early on
|
Discussion
This study aimed to explore and identify positive and negative factors, and factors improving the process of developing project documents, inviting providers and developing design solutions in the CRANE PCP process.
The start-up and phase 1 of the CRANE PCP process have clear elements of following an iterative process design (Ries, 2011). Some of the workshop participants have stated that the capability of change has been good. It was early in the process stated needs of strengthening the technology competence. This was addressed early and improved by including an experienced technological competence in the PCP management team. But there have been elements that could improve this capability, i.e. smoothing the process of changing the scope of work and focus.
Patients were early involved in the process of defining the CRANE solution. This has been important in defining KPI’s for evaluating tenders and later solutions and ensuring a Value-Based development and procurement processes approach.
Complex projects like CRANE needs attention building common level of understanding. Partners from different countries and cultures, and with different experiences is challenging. It’s especially challenging to have the same level of understanding concerning the vision, goals and deliverables. This includes common understanding of the guidelines and requirements from the commission. Attention early on these factors are important to ensure efficiency and quality in complex projects like CRANE. Physical meetings and a clearly defined role of coordination are important for improving the work of complex projects. Physical meetings were difficult at the start of CRANE because of Covid, but more frequent digital meetings and structured meetings would have improved efficiency and quality of the process and deliverables.
Conclusion
It is important to make the process flexible and agile in designing and running complex projects like the CRANE PCP project. Including the process itself, the deliverables, and the organization. Several positive and negative factors have been identified during the early phase of CRANE PCP. The results are structured and used as input for improvements. According to the workshop participants, the most important improvements connected with cooperation and teamwork are:
1. Common level of understanding. Early build a common level of understanding of the overall vision, goal, and focus of the PCP process (
Table 2, no. 1). This includes having more physical meetings (
Table 2, no. 2)
2. Planning and coordination. A few of the input points to the need for better planning and execution of meetings (
Table 2, no. 3). Some of the participants did include the need to improve the project handbook, agreed workplans, fixed meeting series, and more efforts used on coordination (
Table 2, no. 3 and
Table 4, no. 6).
3. Competence. Evaluating the tenders could have been improved by involvement of an external technology-oriented feedback team (
Table 4, no. 7).
The most important improvements connected with the deliverables according to the workshop participants are:
4. Documents. Clarification of requirements, better overview, better handling of documents and better alignment of documents are mentioned in different ways (
Table 4, no. 6). Better guidelines from the commission are also mentioned (
Table 4, no. 7).
5. Developing the documents. A better level of understanding of what to be delivered and who is responsible of what should have been improved (
Table 4, no. 7).
Further research
This lesson learned paper is part of a process identifying elements improving the CRANE PCP process. Corresponding lessons learned workshops will be done throughout the PCP process, and a summary report is planned after the last phase of the process.
Limitations
The outcomes from this study should not be generalized for PCP processes without discussion of content and goal similarities. However, it may give valuable input adapting learnings from the CRANE PCP processes to running similar PCP processes.
Ethics and consent
No ethical approval was required and consent were given orally.
This study is not subject to approval from the Regional Committees for Medical and Health Research Ethics (NO), as it does not contain medical and health research on human beings, human biological material or personal health data. The collected data is not personal health data and was anonymized and transferred in a proper and secure matter to the corresponding author. No patients were participating in the workshop. They were all professionals within healthcare, and CRANE partner representatives.
Participation in the workshops was voluntary, and the collected data was based on an informed consent allowing use for improvements and research purposes. The consent was given orally when the coordinators recruited the participants, and at the start of the workshops. No reservations were made. The consent was given orally because the benefit of collecting and storing signed consent forms was assessed against privacy considerations, and oral consent through conclusive behavior (clear affirmative action) was deemed sufficient pursuit to GDPR Art. 4 i.e. Art. 4 GDPR – Definitions - General Data Protection Regulation (GDPR):
• Oral consent is more appropriate in this workshop setting allowed for easy reservation/opt-out. Oral consent is faster, simpler, less intrusive, improves user experience, and is more suitable when the risk level is low.
• A written consent form may create a higher threshold for participation, signal unnecessary risk, introduce administrative burden, and give an impression of formality disproportionate to the nature of the project.
• Proportionality principle: Measures must be proportionate to the level of risk. A low-risk questionnaire and workshop participation do not require the same level of formality as research involving health data or vulnerable groups. Oral consent was assessed as proportionate and professionally sound.
• Storage limitation principle, cf. GDPR Art. 5 (1)(e) and (c), was upheld since no signed consent forms were stored.
Acknowledgments
This study has been possible thanks to the help and participation from the CRANE partners. Special thanks to the coordinators from region Västerbotten in Sweden, Extremadura in Spain, and Agder in Norway for their insights and help in administrating the process of data collection.
References
- Boston Consulting Group:
Procurement Unlocks Value-Based Health Care.2024.
Reference Source
- Horizon:2020. Reference Source
- Ørjasæter N:
Datafile.2026. Publisher Full Text
- Ørjasæter N, Kværner KJ, Støme LN:
Value Based Framework for Evaluating Pre-Commercial Procurement: Case Study of Value-Based Key Performance Indicators.
J. Med. Internet Res.
2025; 9: e71279. Publisher Full Text
- Ries E:
The Lean Startup. How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.
New York:
Crown Business;
2011.
Reference Source
