Introduction
For decades, the concept of Technology Readiness Levels (TRLs) has been fundamental in assessing the maturity of emerging technologies Mankins (1995). Originating within the National Aeronautics and Space Administration (NASA) in the 1970s to manage risk in complex space missions, the TRL scale provides a standardized language and metric system Sadin et al. (1989). TRL assesses the maturity of a particular technology on a nine-level scale, progressing from basic principles observed (TRL 1) through laboratory validation (TRL 4), demonstration in relevant and operational environments (TRL 6-7), system completion and qualification (TRL 8), up to the actual system being proven in its operational environment (TRL 9). Its primary purpose is to provide a common understanding of a technology’s status, manage development risks, and inform decisions regarding funding allocation and the transition of the technology into systems or towards market deployment. Its utility led to widespread adoption by other major organizations, including the US Department of Defense (DoD), the European Space Agency (ESA), and notably, the European Union (EU) for its research and innovation funding programmes like Horizon 2020 and Horizon Europe, including the European Innovation Council (EIC) Héder (2017).
While widely used, the TRL scale has well-known limitations Olechowski et al. (2017), making it less effective for assessing complex systems or technologies with broad societal or market impacts. For instance, it does not sufficiently address the risks and complexities involved in integrating multiple technologies, including mature ones, into a larger system. Besides, it does not fully consider non-technical factors that are important for innovation deployment. Furthermore, it often does not adequately cover the transition phase between laboratory validation (typically TRL 4-5) and a market-ready system (TRL 8-9). This gap, often referred to as the “Valley of Death” Markham et al. (2010), corresponds approximately to TRLs 4-7, where funding and focus can be insufficient as technologies transition from academic research to private sector development. Holistic frameworks seek to address this by considering factors beyond technical proof-of-concept earlier in the development cycle.
Bridging this gap in civil security involves institutional dynamics that differ from open commercial markets. European R&I funding instruments, such as Horizon Europe Cluster 3 (“Civil Security for Society”), support collaborative technology development with end-users such as Law Enforcement Agencies (LEAs) and first responders. However, a structural bottleneck occurs at project completion: R&I grant mechanisms cannot directly fund public acquisition. Transitioning project outputs into operational capabilities requires public buyers to use formal Pre-Commercial Procurement (PCP) or Public Procurement for Innovation (PPI) pathways (Edquist and Zabala-Iturriagagoitia 2012; Georghiou et al. 2014; OECD 2011). This transition often creates delays due to rigid tendering rules, public sector risk-aversion, and the financial gap between grant expiration and commercial procurement. In parallel, public safety technologies must comply with frameworks on fundamental rights, data protection under the GDPR (European Parliament and Council 2016), and Trustworthy AI guidelines (High-Level Expert Group on AI 2019; Floridi et al. 2018).
Acknowledging the limitations of TRL, several additional Readiness Level (RL) dimensions have been suggested and, in certain instances, implemented to offer a more comprehensive assessment of an innovation’s maturity and likelihood of successful deployment. These RL dimensions address critical non-technical factors such as the readiness of the human-system inter- face (Human Readiness Level, HRL) See (2021), integration (IRL) Sauser et al. (2008), the overall system (SRL) Sauser et al. (2006), manufacturing (MRL) Department of Defense (2023), commercialization, business, and market readiness (CRL, BRL, and again MRL) Gerdsri and Manotungvorapun (2021); Vik et al. (2021), societal acceptance (again SRL or SocRL) Stilgoe et al. (2013), acceptance and regulation (Acceptance RL, ARL, and Regulatory RL, RRL) Vik et al. (2021), organizational and legal alignment (Organizational RL, ORL, and Legal RL, LRL) Bruno et al. (2020), and even technology-specific dimensions such as AI Readiness Level (AIRL) and Data Readiness Level (DRL) for innovations where data is a key resource Eljasik-Swoboda et al. (2019). A more recent addition is Demand RL Paun (2012), a scale proposed to hybridise market-pull with technology-push approaches in technology transfer. Note that the acronym DRL is used for both Data RL and Demand RL; as with MRL above, the intended meaning follows from the context in which the acronym appears. The development of these various RLs, in some cases with the same acronyms but for different concepts, has complicated the landscape, despite addressing TRL limitations. This “Tower of Babel” effect, with multiple terminologies, varying definitions even within the same RL type (e.g., different CRL scales
Australian Renewable Energy Agency (2014); Gerdsri and Manotungvorapun (2021); Bruno et al. (2020)), diverse methodologies, and a lack of universal integration, presents challenges for consistent and efficient holistic assessment. Users must navigate this fragmented environment, potentially increasing the burden compared to the simplicity of the TRL scale Straub (2015).
Additionally, the quantitative aspiration suggested by the numerical scales used in most RL frameworks differs from the qualitative, evidence-based evaluations needed for non-technical fields like SocRL or CRL. This can lead to a false sense of precision or objectivity, masking underlying complexities and subjective judgments, and has led to critiques regarding the mathematical validity of combining ordinal scores Mankins (2009); Olechowski et al. (2017). Many of these complementary dimensions (e.g., Manufacturing RL, CRL, SocRL) incorporate external stakeholder interactions and feedback loops as key elements to show readiness Rip (2014); Stilgoe et al. (2013). This represents a shift from the internally focused technical demonstrations commonly associated with TRL assessment, indicating an understanding that effective innovation depends not only on internal R&D but also on alignment with the external ecosystem, market demands, and societal values.
In this context, the EU-funded MultiRATE project aims to create a holistic RL assessment framework to evaluate the holistic maturity of products, systems, or processes developed in civil security R&I projects for both LEAs and non-LEAs. MultiRATE’s white paper van der Lee et al. (2025) introduced its components, the design process that they were considering at the beginning of the project and discussed insights for its implementation. The RL dimensions include adapted versions of existing RL scales like TRL, SocRL, IRL, CRL, MRL (Manufacturing) as well as additional RL dimensions relevant in the field, such as Security RL (SecRL) and Legal, Privacy, and Ethics RL (LPERL) Adomaitis et al. (2024). All these dimensions needed to be holistically harmonized to meet the project’s objectives. Additionally, the framework includes an investment forecasting module to estimate the costs required to increase RL maturity.
Holistic readiness assessment is especially relevant in the EU civil security research and innovation landscape. Under Horizon Europe, the EU’s research and innovation framework programme for 2021–2027 (European Commission 2021), security research is supported through the dedicated “Civil Security for Society” cluster, which funds projects aimed at protecting citizens, society, and infrastructures from a wide range of threats while complying with EU values and fundamental rights (European Commission 2022). Technologies developed in this context must often reconcile technical maturity with societal acceptance and legal and ethical compliance — the dimensions captured by SocRL, LPERL, and SecRL in the MultiRATE framework — which makes a holistic assessment approach particularly relevant for the sector. The uptake of results from such projects — a long-standing concern in EU security research policy (van der Lee et al. 2025) — depends on maturity dimensions that go well beyond technical readiness, which motivates sector-specific assessment frameworks such as the one proposed in this work.
This paper elaborates on the implementation of this holistic framework and the evaluation of its feasibility by the project’s conclusion. More specifically, this paper contributes to the state of the art in holistic readiness assessment in the following ways:
• The MultiRATE Holistic Readiness Level (HRL) calculator that integrates the indicators and calculations from seven RL dimensions taking into account all their peculiarities adapted to the civil security sector. The web-based MultiRATE HRL calculator can be accessed here: https://hrl.multiratehl.iti.gr
• The adaptation and harmonization process to the security sector of the seven RL dimensions of the MultiRATE HRL: TRL, SocRL, SecRL, LPERL, IRL, en-CRL, and MRL.
• The integration process of an investment forecasting module to effectively estimate the costs required to increase RL maturity in the context of the MultiRATE HRL.
• The validation process of the MultiRATE HRL and its components with the collaboration of an external network of experts following the Design Science Research Methodology (DSRM) Peffers et al. (2007).
Related work
Setting up a conceptual framework that moves from a disjointed collection of independent one-dimensional RL scales toward an integrated, actionable, truly holistic viewpoint is still the main challenge in addressing the complex nature of innovation.
Table 1 provides a comparative overview of the RL dimensions that form the building blocks of most prominent state-of-the-art holistic approaches.
Table 1. Comparison of RL dimensions, complementary to TRL, forming the basis of most prominent state-of-the-art holistic frameworks.
| Framework | Primary focus | Strengths | Weaknesses |
|---|
| TRL Mankins (1995) | Technical feasibility of a single component. | Simple, ubiquitous, common language for technical maturity. | Ignores integration, manufacturing, market, and societal factors. Poor predictor of overall success. |
| SRL/IRL Sauser et al. (2006) | Technical feasibility and integration maturity of a system. | Forces rigorous architectural analysis and focus on interfaces. | Mathematically questionable (ordinal data), complex, lacks formal adoption outside defence/aerospace. |
| AIRL/DRL Eljasik-Swoboda et al. (2019) | Readiness of algorithms and data for AI/ML systems. | Adapts RL concepts for data-centric innovations, acknowledging data as a key resource. | Highly specific to a technology type, not an application domain; less standardized than other RLs. |
| ORL/LRL Bruno et al. (2020) | Organizational change management and legal/regulatory compliance. | Critical for deployment in regulated industries or public sector environments. | Highly context-specific; can be difficult to assess proactively before policies are set. |
| MRL Department of Defense (2023) | Producibility and manufacturing risk. | Structured roadmap for de-risking production; links engineering to manufacturing. | Resource-intensive, requires specialized expertise, limited adoption outside of defence/manufacturing. |
| Market RL /CRL/BRL Vik et al. (2021);
Australian Renewable Energy Agency (2014); Gerdsri and Manotungvorapun (2021) | Commercial viability, market/business case maturity, and commodification. | Focuses on market-pull and investment-potential from an early stage. | Lack of a single standard leads to inconsistency; can be subjective. |
| SocRL Stilgoe et al. (2013) | Societal acceptance, ethical implications, stakeholder engagement. | Addresses critical non- technical barriers to adoption for impactful technologies. | Can be highly subjective and difficult to quantify; newer and less validated than other RLs. |
| Human RL See (2021) | Human-system interaction, usability, and training. | Ensures technology is effective and safe in the hands of users, filling a major TRL gap. | Requires human factors expertise; can be subjective and resource-intensive to test properly. |
| Demand RL Paun (2012) | Market-pull and technology-push hybridisation for technology transfer. | Supports demand-side assessment in technology transfer practices, complementing technology-push approaches. | Overlaps with Market RL; acronym collides with Data RL (see text). |
| Acceptance RL/RRL Vik et al. (2021) | User acceptance and regulatory/legal alignment. | Addresses social and political aspects of innovation that can be major barriers to deployment. | The concepts are less standardized than others and are often highly context specific. |
The evolution of RL assessment beyond TRL can be seen as a series of reactive expansions, each designed to address a critical RL dimension over- looked by its predecessors Straub (2015). The development of SRL/IRL addressed technical system-level concerns, with a rigorous analysis of integration risks. However, its methods have been critiqued for performing arithmetic operations on ordinal data—ranked scales where the intervals are not uniform—which can create a false sense of precision Olechowski et al. (2017). Following with technology-specific needs, AIRL and Data RL adapt readiness concepts for data-centric innovations where the quality and availability of data are paramount. Likewise, dimensions like ORL, LRL, and RRL emerged to assess readiness for deployment in complex organizational and regulatory environments, but their context-specific nature makes them difficult to apply proactively before policies are set.
On the other hand, the creation of Market RL and CRL/BRL expanded into the socio-economic sphere. These RL scales focus on market-pull and commercial viability, though the lack of a single standard for these can lead to subjective and inconsistent assessments. It is important to disambiguate MRL, as the acronym has been used to denote both Manufacturing Readiness Level, which provides a structured roadmap for de-risking production Department of Defense (2023), and Market Readiness Level, as defined within the Balanced Readiness Level assessment (BRLa) framework Vik et al. (2021). Henceforth, the acronym MRL will be used exclusively to denote Manufacturing RL.
Another expansion moved into the human and societal context. SocRL, Human RL, and Acceptance RL address public acceptance, usability, and user acceptance respectively, but all require significant qualitative analysis that can be difficult to standardize and resource-intensive to conduct. These complementary RL scales also share common challenges, as noted in
Table 1, like that they are often resource-intensive, difficult to quantify, and lack universal standardization, motivating the development of holistic frameworks to manage their complexity.
Current state-of-the-art holistic frameworks can be broadly categorized by their primary philosophy, as reflected in
Table 2. These frameworks offer different strategic approaches to holistic assessment. The US DoD’s System Readiness Assessment (SRA), yielding an SRL, is a prime example of a “composite index” approach that attempts to create a single, unified score Sauser et al. (2006). For developing complex security systems, its strength is the rigorous focus on integration, which is critical for ensuring interoperability. However, its methodology of calculating a single, composite score has been critiqued for its use of arithmetic operations on ordinal data, which can obscure critical weaknesses in individual dimensions and may be less suited for the nuanced trade-offs required in security R&I Olechowski et al. (2017). In contrast, other frameworks focus on standardizing the assessment “process”. The GAO’s TRA guide, for example, offers a best-practice approach to carrying out TRAs and formulating Technology Maturation Plans for them U.S. Government Accountability Office (2020). To some extent, this is because it treats the process as an evidence-based, systematic one. However, it is not a multi-dimensional scale itself and remains heavily centred on the TRL metric, thereby not fully solving the problem of a narrow technical focus.
Table 2. Comparative overview of state-of-the-art holistic readiness frameworks.
| Framework | Core purpose | Key dimensions assessed (beyond TRL) |
Limitations/Critiques |
|---|
| Composite Index Approach
|
| SRA/SRL Sauser et al. (2006) | Assess system-level maturity and risk by combining technology and integration readiness. | IRL | Mathematical validity of combining ordinal data is questioned; potential for misleading results; still largely experimental. |
| Process-Focused Approach
|
| GAO TRA Guide U.S. Government Accountability Office (2020) | Standardize a high-quality, evidence-based process for conducting TRAs and managing risk. | Implicitly considers evidence quality, maturation planning. | Primarily a process guide, not a multi-dimensional RL scale itself; relies heavily on TRL as the core metric. |
| Context-Specific Dashboards
|
| EU 4-Axis Framework Bruno et al. (2020) | Assess the impact and potential of digital technologies for European public services. | SocRL, ORL, LRL | Proposed framework; needs validation outside digital public services; omits commercialization and manufacturing dimensions. |
| Adoption RL Framework U.S. Department of Energy (2023) | Assess non-technical commercialization risks and identify barriers to private sector uptake. | Value Proposition, Market Acceptance, Resource Maturity, License to Operate. | Specific to commercialization; may overlap with CRL/MRL; newer framework still gaining adoption. |
| BRLa Vik et al. (2021) | Provide a balanced, qualitative assessment of emerging technologies using a dashboard approach. | Market RL, RRL, Acceptance RL, ORL | A diagnostic tool, not a prescriptive one; dimensions are generic and require adaptation for specific domains. |
| IDE Startup Framework Gerdsri and Manotungvorapun (2021) | Map a pathway toward sustainable growth by connecting technical and business readiness. | MRL, CRL, BRL | Context-specific to startups; primarily focused on a commercialization pathway. |
| READINESS-Navigator Eljasik-Swoboda et al. (2019) | Adapt readiness assessment for AI/ML innovations where data is a key resource. | Replaces MRL with AIRL and DRL. | Highly specific to a technology type; not tailored for an application domain like security. |
| MDRL Framework van de Loo et al. (2022) | Tailored assessment for high-risk medical devices, integrating human factors. | Human RL concepts (usability, comfort), safety, clinical effectiveness. | Highly domain-specific; requires extensive clinical and regulatory expertise. |
A third category, which most closely aligns with the philosophy of MultiRATE, involves context-specific “dashboards” that visualize multiple dimensions. These have been adapted for specific business contexts, technology types, and application domains. For instance, a framework designed for Innovation-Driven Enterprise (IDE) startups explicitly connects TRL and Manufacturing RL with Commercialization and Business RLs (CRL and BRL) to map a pathway toward sustainable growth Gerdsri and Manotungvorapun (2021). For assessing AI-based innovations, the READINESS-Navigator tool adapts the readiness concept by replacing MRL with specific AI and Data RLs (AIRL and DRL), acknowledging that data is the critical raw material Eljasik-Swoboda et al. (2019). The EU 4-Axis framework (TRL, SocRL, ORL, LRL) is another key example tailored to the application do- main of public sector innovation Bruno et al. (2020). While its inclusion of organizational and legal aspects is highly relevant, it omits the commercialization and manufacturing dimensions that are vital for ensuring a technology can be produced and sustained as a viable product. Similarly, the US Department of Energy’s Adoption RL (ARL) framework is tailored to energy commercialization, focusing on a technology’s value proposition and market acceptance U.S. Department of Energy (2023), and the Balanced Readiness Level Assessment (BRLa) uses generic dimensions (e.g., “Regulatory”, “Acceptance”) for agricultural technology that lack the specificity required for the security domain Vik et al. (2021). A notable example in the medical field is the Medical Device RL (MDRL) framework, which integrates Human RL concepts to assess not only technical function but also usability, comfort, and patient response van de Loo et al. (2022).
Finally, it is worth noting the existence of highly detailed holistic maturity models designed for the specific context of cybersecurity, such as the ENISA CSIRT framework
European Union Agency for Cybersecurity (ENISA) (2019) and the Cybersecurity Capability Maturity Model (C2M2) U.S. Department of Energy (2022). Unlike the technology-focused frameworks in
Table 2, these models assess the maturity of an entire “organization’s” capabilities, processes, and security posture. Methodologically, they are instructive as they demonstrate how a complex domain can be broken down into granular components (e.g., C2M2’s use of domains and objectives; ENISA’s parameters for Organization, Human, Tools, and Processes) for detailed assessment. However, their purpose—assessing organizational posture—is fundamentally different from the goal of assessing a specific technology’s readiness for uptake, which is the focus of MultiRATE.
In summary, existing holistic frameworks provide valuable philosophical and methodological precedents. The clear evolution toward context- specific, multi-dimensional dashboards aligns with the approach taken by MultiRATE. By visualizing multiple readiness levels, these methods offer a more transparent and actionable alternative to opaque composite indices, allowing for the identification of specific weaknesses. This is a significant advantage over simpler process-focused guides or singular RL scales. However, no existing framework provides the specific combination of harmonized dimensions required for the civil security domain. The state-of-the-art lacks a framework that concurrently and systematically evaluates TRL, IRL, MRL, CRL, SocRL, and, most critically, the specific requirements of SecRL and LPERL Adomaitis et al. (2024). This identified gap calls for a constructive research approach aimed at creating a novel solution. For this purpose, the Design Science Research Methodology (DSRM) provides a suitable structure for developing and evaluating a new artifact to address a specific problem Peffers et al. (2007). Unlike purely descriptive methods that analyse existing phenomena, DSRM offers a rigorous process for the design, implementation, and validation of a practical solution. Its iterative nature is particularly advantageous for a complex undertaking like the MultiRATE HRL, which requires harmonizing multiple, disparate readiness dimensions into a single, coherent framework validated by expert stakeholders. This methodological foundation ensures that the resulting artefact is not only theoretically sound but also demonstrably useful in its intended context.
Method
The methodological approach of the MultiRATE HRL assessment, first introduced in its white paper van der Lee et al. (2025), integrates seven distinct RL dimensions (TRL, SocRL, SecRL, LPERL, IRL, en-CRL, and MRL) to create a holistic evaluation. A key design principle is to maintain the integrity and unique characteristics of each RL scale while providing a holistic view of a technology’s feasibility and progress. To achieve this with heterogeneous RL dimensions, some of which do not share the same number of levels, the MultiRATE HRL adopts a flexible harmonization strategy. This approach is conceptually similar to state-of-the-art holistic frameworks like the IDE Startup Framework Gerdsri and Manotungvorapun (2021), the ENISA CSIRT model
European Union Agency for Cybersecurity (ENISA) (2019), and the C2M2 framework U.S. Department of Energy (2022), which harmonize the levels of different RL scales from a higher-level perspective. This contrasts with the more rigid, one-to-one level mapping found in models like the EU 4-Axis Framework Bruno et al. (2020) and BRLa Vik et al. (2021). The chosen methodology offers the advantage of accommodating diverse RL scales and allowing for the assessment of non-technical maturity improvements required to achieve a ready-to-market solution, even beyond the scope of TRL9.
Another key design principle is to consider two kinds of assessments: (1) a quick holistic RL assessment of an “element” (i.e., the product, system, or process under evaluation) to get a preliminary result, and (2) a full holistic RL assessment of the same element with more time to get a more detailed result. The full holistic assessment is intended to be conducted by a group of expert users that would answer the full list indicators related to their expertise. The result of this full assessment will be the most accurate and detailed that MultiRATE HRL can provide. On the other hand, the quick holistic assessment is intended to be conducted by only one user, not necessarily an expert in all the fields. In this case, the result will not be as accurate and detailed, but it will be quicker and simpler to answer by a non-expert. Therefore, in practice, the quick assessment modality could be seen as a preparation for a full assessment, while the full assessment modality is required for a formal assessment with more detailed insights on the obtained MultiRATE HRL and the more specific pending points towards a higher maturity of the element. Results from a quick assessment should be interpreted as a screening approximation rather than as a formal evaluation, and should not be used alone for consequential decisions (e.g., funding or procurement). The distinction between expert and non-expert judgement is well documented: expertise develops through extended experience in a domain and does not transfer reliably across domains Kahneman and Klein (2009); Dreyfus and Dreyfus (1980). In particular, for dimensions that require specialized expertise — such as LPERL, SecRL, and the qualitative aspects of SocRL and en-CRL — quick assessment results should be confirmed through a full assessment conducted by the relevant domain experts before being used in decision-making contexts.
To measure the MultiRATE HRL of an element, the user must answer a series of indicators associated to each of the RL dimensions. In a full assessment, the indicators of TRL, SocRL, SecRL, IRL, en-CRL and MRL can be answered with five possible answers: “N/A” (Not applicable), “Nothing”, “Partially”, “Largely” and “Fully” implemented. If all indicators related to a level of the RL scale are answered with “Fully”, the level is 100% achieved, if all are “Nothing”, the level is 0% achieved, and the in-betweens are calculated proportionally. The “N/A” responses are ignored from the computation, but there is a limit of N/A responses per level to be able to compute the results (20%). In the quick assessment, as there are a few questions per level, N/A is not available as a possible answer. LPERL Adomaitis et al. (2024) is computed in a different way as explained in the following subsection, with “Yes”/“No” possible answers. Next, we will introduce more in detail the individual MultiRATE RL dimensions and then how they are combined for a holistic assessment.
MultiRATE readiness level dimensions
a) Technology Readiness Level (TRL)
The TRL assessment methodology developed within the scope of MultiRATE, is mainly based on the categories of indicators that were defined in previous attempts and builds on them, according to the further research which has been conducted within MultiRATE. Those indicators form the base, on the list of questions which apply for each TRL level. Under MultiRATE project, TRL further expanded the TRL assessment methodology enhancing the developed tool in order to further improve the quality of the assessments. To enrich the framework, an extensive literature review has been conducted, resulting in the extraction and inclusion of additional indicators while the indicators were further modified and tailored to fit the usage of the overall approach to the EU R&D projects. These indicators provide a more comprehensive and context-specific evaluation, ensuring that the framework captures relevant factors for assessing the system’s readiness and usability.
The evaluation framework of the MultiRATE TRL assessment, comprises four distinct indicator categories which include all the identified indicators: 1) Technology Preparation & Requirements, 2) Documentation, 3) Operability & Continuity, and 4) Evaluation & Usability. These categories serve as key aspects for assessing the readiness and effectiveness of a system. Each RL is comprised by a set of indicators. Some indicative aspects of indicators per category include the following as indicated in
Table 3.
Table 3. MultiRATE TRL indicator categories.
| Category |
Indicative aspects of indicators included |
|---|
| Technology Preparation & Requirements | User requirements, system specifications, architecture, data flows, HW and SW requirements, software specifications |
| Documentation | Documentation of specifications, installation and operation guides, user guides |
| Operability & Continuity | Deployment, installation, operation, and maintenance |
| Evaluation & Usability | End user testing and evaluation, overall efficiency of the tool, end-user friendliness |
b) Societal Readiness Level (SocRL)
The MultiRATE SocRL is used to assess the level of adaptation of an innovation to be successfully adopted by society. The use of the umbrella term ‘innovation’ in the case of the SocRL is a purposeful indication of the necessary focus it has for its assessment. Innovations include the use of a new piece of equipment, system, software, methodology, or procedure within a context it has not been used before. Innovations are based on the results of new technological developments, technology combinations, or other knowledge towards societal well-being, and should be developed alongside continuous stakeholder input to ensure they are suitable for their intended purpose and accepted within their context. The inclusion of a variety of different stakeholders in the research and innovation process forms a key component to validating readiness as part of the SocRL assessment.
Previous work has been carried out into societal readiness, with the most well-known scale coming from Innovation Fund Denmark Innovation Fund Denmark (2019). However, the existing works on societal readiness were fragmented and incomplete for the purposes of MultiRATE’s proposed comprehensive assessment framework; additionally, past research tended to focus on the areas of energy, decarbonization, and sustainability Sprenkeling et al. (2022); Büscher et al. (2023), and therefore more work was needed in order to apply these societal considerations to the security domain, where they are just as, if not more, relevant than ever.
The MultiRATE SocRL includes measures such as the take-up and acceptance of an innovation by society. There are four categories of indicators that together form the criteria for the assessment: 1) Stakeholder Engagement, 2) Acceptance, 3) Societal Integration, and 4) Societal Good, presented in full in
Table 4. These categories and the indicators associated with them are based on the work of past tools used to assess societal readiness in a variety of different ways Space53; New Horrizon; IsITethical. The SocRL features nine levels, ranging from the initial identification of the societal need, societal good, and associated readiness aspects of the innovation in question, all the way up to finally proving its benefit within society after launch on the market. The levels used in the MultiRATE assessment framework are adapted from those previously proposed for societal readiness Sprenkeling et al. (2022); Büscher et al. (2023).
Table 4. MultiRATE SocRL indicator categories.
| Category | Indicative aspects of indicators included |
|---|
| Stakeholder Engagement | Engaging and involving a wide range of stakeholders and their considerations in the research and innovation process. |
| Acceptance | Taking into account the benefits, risks, societal concerns, and trust related to the innovation and involved practitioners through stakeholder engagement, especially with citizens. |
| Societal Integration | How the innovation fits into practices, place-based policy, cultures, and social trends, helping ease the integration of the innovation into everyday lives of the future. |
| Societal Good | The intentions, priorities, co-benefits, and consequences of the innovation, contributing to and promoting societal well-being. |
The SocRL predicts the readiness of and helps prepare an innovation along its journey of adoption. The full assessment has a total of 50 indicators, however, a quick version of the SocRL assessment was also developed for MultiRATE that has only 14 indicators across all nine levels. The quick assessment works by using only a handful of the most crucial indicators across the different categories and levels. The result is a highly reduced assessment, but one that can still give an approximation of the user’s potential SocRL without undertaking a full assessment.
c) Security Readiness Level (SecRL)
The MultiRATE SecRL scale allows assessing the degree of security of a given element and its assets in relation to its threat environment and the applied security measures as well as the security testing complexity. The SecRL is a scale for measuring the security of an innovation, product, or solution (a process or a technology) and its assets concerning its threat environment. Each level increase implies risk reduction, meaning reducing the likelihood and severity (e.g., damage) of the impact of security incidents by adjustments in design and introducing more or better countermeasures or security capabilities. Since there was no previous attempt for the SecRL that aligns the MultiRATE’s purpose of evaluating and determining the security maturity of any kind of solution (in the field of civil security), it had to be built within the scope of MultiRATE. The methodology used was to harmonize the previously identified Security Maturity Model with international security standards such as ISO 15408. The basic framework and level structure are derived from the guidelines for using TRLs in ESA programs. It ranges from SecRL 1 (security considerations) as the lowest level to SecRL9 (Operational Security Validation) as the highest level. The total number of relevant indicators for the nine security levels has been streamlined to 41. There are eight different types of indicators, which provide valuable insights into various aspects of security readiness. Among these types, the largest one is the “Security Testing, Improvement, and Iteration” type, which encompasses most indicators ranging from SecRL 5 to SecRL 8. Although there was a possibility to further divide the “Security Testing, Improvement, and Iteration” type into more detailed subtypes such as “test development,” “test conduction,” and “test iteration,” the decision was made to maintain good alignment with the other RLs in MultiRATE. This ensures consistency across the different readiness levels and facilitates comparison and analysis. The externalization of definitions, explanations, and examples helped to make the indicators shorter and more precise. Further details can be found in Adomaitis et al. (2024).
d) Legal, Privacy, and Ethics Readiness Level (LPERL)
The MultiRATE LPERL scale allows assessing the degree of alignment with European values and rights, provide guidance in ethics by design and promote responsible application of solutions. A key aspect is the identification- characterization-harmonization-control paradigm. The four-level structure of LPERL applies this paradigm to the domain of technology ethics, providing a systematic approach to identifying, characterizing, harmonizing, and taking control of ethical issues.
LPERL 1 encompasses the step of identifying potential ethical dilemmas, privacy concerns, and legal implications associated with the system. At LPERL 2, the relationships and trade-offs between various ethical issues are reflected. LPERL 3 shows operationalization and involves the integration of ethics considerations into the system design, a process often referred to as “ethics by design”. Finally, LPERL 4 shows control mechanisms for accountability and the attainment of standard benchmarks and certifications, if applicable. This ERL structure combines anticipatory ethics with ethics by design and streamlines the management of emerging AI technologies.
To evaluate LPERL level, the tool includes more than 200 indicators grouped in blocks. In the area of security, the LPERL blocks include: 1) a basic technology ethics module for all use cases, 2) AI-specific block, 3) Law Enforcement block, 4) Personal data block. The number of indicators actually used during evaluation depends on the use case. Further details can be found in Adomaitis et al. (2024).
e) Integration Readiness Level (IRL)
The Integration Readiness Level (IRL) scale is a framework for determining the maturity and readiness of an element for integration into a larger system or environment. It provides a standardized method for assessing the readiness of technologies, software, or hardware for integration, assisting in the identification of potential risks and issues and promoting effective decision-making during the integration process by establishing clear benchmarks and criteria for assessing readiness. Furthermore, it allows stakeholders to track the status of integration efforts, prioritize activities, and allocate resources more efficiently reducing integration risks by encouraging early detection and resolution of difficulties, hence enhancing the overall success rate of integration initiatives.
The IRL framework consists of 34 indicators grouped into nine levels. Originally based on the Sauser and Forbes model, the IRL framework has been refined to provide a more systematic and progressive approach. This updated model guides end-users through the integration process, from initial identification to real-world applicability. The 9 level descriptions are presented in
Table 5.
Table 5. Description of the MultiRATE Integration Readiness Levels (IRL).
| IRL level | Description |
|---|
| 1. High-level concepts identified | Fundamental concepts and overarching principles relevant to the integration process have been identified and delineated. These involve defining high-level goals, outlining the overall architecture, identifying critical components, and establishing initial approaches to integration. |
| 2. Basic requirement specifications defined | The essential functionalities, constraints, and criteria that the integrated solution must fulfil to meet its intended objectives are defined. |
| 3. User requirements and integration tests defined | The user requirements that the integrated solution will meet are clear and a framework for the integration testing is defined. |
| 4. Technology validated in laboratory environment | The functionality, performance, and feasibility of integrating the elements have been validated in a controlled laboratory environment. |
| 5. Technology validated in relevant environment | The performance, functionality, and compatibility of the elements have been validated in an environment that closely resembles the conditions in which it will be deployed and used in real-world scenarios. |
| 6. Technology validated End-to-end operations | The integrated elements were tested and validated across its entire operational workflow, from start to finish, under real-world conditions. |
| 7. System prototype demonstrated in operational environment | A functional system prototype has been demonstrated in a operational environment with a tangible demonstration of its capabilities, performance, and suitability for real-world deployment. |
| 8. System integration validated by its end-users
| The integrated system of the element is tested with end-users in the operational environment and is then prepared for long-term maintenance. |
| 9. System integration proven in operational environment | The system is deployed in a operational environment and the monitoring/maintenance procedures are in place. |
To this end, the IRL follows a structured progression via three key stages: (i) The definition and specification; (ii) the testing and validation and (iii) the deployment and optimization stage. In the first stage key elements are identified, objectives are set, and specifications are clearly defined to establish a solid integration framework. The second phase focuses on testing and validation to ensure successful integration in different context environments. In the last stage, the system is deployed in real-world conditions, monitored for stability, and continuously optimized to ensure long-term reliability and performance improvements.
In order to facilitate the users’ assessments, a simplified version of the IRL framework has also been designed. This simplified scale retains the most representative indicators, allowing end-users to conduct assessments more efficiently maintaining the objectives of the original framework.
f
) Enhanced Commercialization Readiness Level (en-CRL)
The MultiRATE en-CRL scale enhances previous CRL models Paun (2012);
Australian Renewable Energy Agency (2014); Vik et al. (2021); Gerdsri and Manotungvorapun (2021) to align with the Holistic Innovation Management Methodology (HIM) Sofou (2017). Following the HIM, the scale builds on project “Results” -as these are defined in the EU Grant Agreement signed between the European Commission and the consortium of an EU-funded project (i.e., any output of the action, tangible or tangible, whether or not it can be protected)- and aims to improve the commercialization strategy for the products embodying these “Results”.
The en-CRL scale maintains 9 levels suggested in previous work, but differs substantially in that it introduces a list of different indicators for each Level. It allows assessing the maturity of an element to be launched, taking under consideration: the market & competition analysis relative to the element, its IP protection and freedom to operate, the team’s expertise, certifications & adherence to standards, contracts with suppliers, exploitation agreements, customer support plans, as well as marketing & risk mitigation plans. Its goal is to support an exploitation team in assessing their product’s readiness for the market, and enrich the action plan to improve the product’s commercialization strategy and innovation adoption. This is in line with the combined strategy for IP management, exploitation, dissemination and data management of project “Results”, throughout their lifecycle, as proposed in Pestana and Sofou (2024).
Six indicator categories (IP Management, Market Potential and Competitive Landscape, Team and Consortium Expertise, Solution Definition/Design/Development, Exploitation Plan, Manufacturing/Supply Chain) are examined in a 9-step assessment process, and the indicators serve as guidelines to achieve the goal. The full version of en-CRL comprises of 9 Levels, 6 categories and 35 indicators. A version for quick evaluation is also available, with 18 indicators covering all 6 categories and 9 levels.
g) Manufacturing Readiness Level (MRL)
The MultiRATE MRL scale allows assessing the maturity of a manufacturing process throughout its conception, proof of concept, realization in a laboratory scale and demonstration in a pilot production line or a low-rate initial production United States Department of Defense (2025). MRL is used to provide decision makers with a common understanding of the relative maturity and attendant risks associated with manufacturing technologies, products, and processes being considered. It aims to define the current level of manufacturing maturity, to identify maturity shortfalls and associated costs, and to provide the basis for manufacturing risk management. This scale is based on prior MRL models Wheeler and Ulsh (2009); Schumacher et al. (2016); Basu and Ghosh (2017). The original indicators served as a baseline and were subsequently refined, adapted, and expanded to address the specific context and aims of EU Research and Development Security Projects. These indicators provide a more comprehensive and context-specific evaluation, ensuring that the framework captures relevant factors for assessing the system’s manufacturing readiness. The evaluation framework of the MultiRATE MRL assessment, includes 67 total indicators, consists of 9 levels of Manufacturing Maturity, comprises 8 distinct indicator categories namely: Manufacturing Process, Supply Chain Readiness, Production Capacity, Quality Control and Assurance, Manufacturing equipment and machinery, Cost and Efficiency, Training and Workforce, Regulatory and Compliance.
MultiRATE holistic readiness level assessment
The holistic RL assessment workflow consists of three main phases: (1) Initialization, (2) HRL calculation and (3) HRL results analysis. Next, we will describe each of these phases more in detail.
a) Initialization
During the initialization phase, the user(s) of the HRL assessment system need to select some values from a series of concepts shown in
Table 6 to configure the system for a specific assessment.
Table 6. Concepts in the initialization form.
| Concept name | Options |
|---|
| Security area | Border management, Disaster Resilient Societies, Fighting Crime and Terrorism, Digital Security, Infrastructure Resilience, Strengthened Security Research and Innovation, Other |
| Nature of the element | Hardware, Software, Hardware&software, Methodology/Procedure, Methodology/Procedure&Hardware, Methodology/Procedure&Software, Methodology/Procedure&Hardware&Software |
| Involves human subjects | Yes/No |
| Includes AI | Yes/No |
| Use case | Tech development orientation, Innovation uptake (sup- ply side), Innovation uptake (demand side), Standardization policy support, R&I programming and management support |
| User type | LEA (Law Enforcement Agency), Other first responder, Critical entity, Technology supplier, Other |
| Purpose | Self-assessment, Assessment by another company, Assessment by a 3rd party for certification |
| Quick assessment | Yes/No |
| Starting xRL | Excluded/1-9/Complete |
| Exclude LPERL | Yes/No |
These concepts include the security field area to which the project belongs, whether the element involves human subjects, and if it includes AI. To aid in understanding the context of the users who will perform the assessment, there are concepts related to the specific use case, user type, purpose of the assessment, and whether it is a quick or full assessment. Additionally, the final concepts assist in refining the starting point of the RL scales or excluding them from the assessment. In the case of LPERL, there is no starting point as previously explained, so it only has the option of being excluded or not.
b) HRL calculation
After the initialization, users can access relevant indicators of the RL dimensions to assess an element’s holistic maturity. As explained at the beginning of this section, by responding to these indicators, the degree of accomplishment for each level is calculated as a percentage. A level is considered “Fully Achieved” if the percentage is above 80%, “Partially achieved” if 60-80% and “Not Achieved” if below that. There are also some additional rules to be considered during the HRL assessment. If a level is partially achieved, the next level in that dimensions can be assessed even if it is not accomplished. Another rule is that if a level is partially achieved, the following one can be fully achieved but the next one cannot be assessed until the previous partially achieved levels grow to fully achieved.
Thus, the user can visualize the results with a multi-dimensional RL dashboard in a similar way to other state-of-the-art holistic RL frameworks. However, our approach also integrates the RL dimensions into four holistic RLs that numerically represent the overall RL during the development of an element until it is ready for the market: HRL1—Solution concept, HRL2—Solution prototype, HRL3—Functional solution and HRL4—Ready- to-market solution. These holistic RLs take the well-established TRL scale as the main reference for the alignment of all RL dimensions into the four holistic RLs. TRL3 “Experimental proof of concept” is aligned with HRL1, TRL6 “Technology demonstrated in relevant environment(s)” is aligned with HRL2, and TRL9 “Actual solution proven in its operational environment” is aligned with HRL3. Beyond TRL9, dimensions that require additional effort for HRL4 include: SocRL7-9, SecRL9, en-CRL8-9, and MRL9. IRL is fully aligned with TRL, while LPERL’s evaluation occurs at each HRL transition, ranging from levels 1 to 4.
Table 7 shows this alignment more in detail.
Table 7. Alignment among MultiRATE RL dimensions for HRL computation.
| HRL | LPERL | SocRL | CRL | TRL | IRL | SecRL | MRL |
|---|
| HRL 4 Ready-to-market solution | 1-4
| 7-9
| 8-9
| 9
| 9
| 9
| 9
|
| HRL 3 Functional solution | 1-4
| 6
| 7
| 7-9
| 7-9
| 7-8
| 7-8
|
| HRL 2 Solution prototype | 1-4
| 4-5
| 4-6
| 4-6
| 4-6
| 4-6
| 4-6
|
| HRL 1 Solution concept | 1-4
| 1-3
| 1-3
| 1-3
| 1-3
| 1-3
| 1-3
|
Each HRL degree of accomplishment is calculated as the average of the aligned levels and that of LPERL. This average is computed on the degrees of accomplishment (percentages) of the aligned levels, which in turn derived from the proportion of indicators answered as “Fully” implemented in each level, as described above; it is therefore an indicative, transparency-first aggregation of the dimension results rather than an arithmetic operation on the ordinal level values themselves. The four holistic levels are always presented together with the full dimension-level breakdown, so that the underlying ordinal information remains visible and weak dimensions can be identified. Therefore, HRL1 computes the average of TRL1-3, SocRL1-3, SecRL1-3, IRL1-3, en-CRL1-3, MRL1-3 and LPERL1-4, HRL2 the average of TRL4-6, SocRL4-5, SecRL4-6, IRL4-6, en-CRL4-6, MRL4-6 and LPERL1-4, HRL3 the average of TRL7-9, SocRL6, SecRL7-8, IRL4-6, en-CRL7-9, MRL7-8 and LPERL1-4, and finally HRL4 the average of TRL9, SocRL7-9, SecRL9, IRL9, en-CRL8-9, MRL9 and LPERL1-4.
c) HRL results analysis
To better understand the obtained HRL results, we recommend using two kinds of graphs: (1) A spider graph including reference RL dimension distributions per holistic level (
Figure 1), and (2) bar graphs showing the degrees of accomplishment (
Figure 2). These graphs allow spotting easily the weak points in the maturity of the element being assessed. More specifically, the spider graph shows the levels achieved (partially or fully) in each dimension compared to the expected achievements for each holistic level, making it easier to identify dimensions requiring further improvement to increase holistic maturity. On the other hand, the bar graphs provide a detailed visualization of the degrees of accomplishment across all levels, regardless of whether they have been fully achieved, allowing users to concentrate on specific dimension levels needing enhancement. Furthermore, by identifying these weak points against the references, a simple program can be developed to automatically provide feedback to the user(s), guiding them towards the next steps in improving the holistic maturity of the element being evaluated.
Figure 1. An example of assessment results visualized with a spider graph with respect to levels expected for each HRL.
Figure 2. An example of assessment results visualized with bar graphs showing degrees of accomplishment.
d) Readiness Level Investment Forecasting
One important objective of the MultiRATE project’s forecasting module was to develop a robust and objective method for estimating the necessary investment for the development of a specific element. Recognizing the inherent variability and lack of balance in monetary investment across different regions, such as within the European Union, we opted to use effort allocated in EU-funded projects, measured in person-months (PMs), as a more stable and representative metric. This approach allows to address the fundamental question: “How much effort, in person-months, is required for an element to advance from one specific Technology Readiness Level (TRL) to another?”
To enable this estimation, the first step is to define an element through meaningful and precisely measurable features. Through extensive internal discussions within the MultiRATE project consortium, it is established that these features must be quantifiable, consistent, and verifiable to build a reliable dataset for the FM system. The discussion focused on data points that have undergone rigorous validation by the European Commission (EC), ensuring their accuracy as part of either funded or completed projects. The final set of features selected for our analysis includes:
•
Project Duration: The total time span of the project in which the element was developed, and the duration of the element development.
•
Technology Readiness Levels (TRLs): Both the initial and final TRLs of the element, indicating its developmental stage.
•
Person-Months (PMs): The total effort expended on the project, and specifically on the element’s development, measured in person- months.
•
Number of Participants: The total number of organizations or individuals involved in the project and specifically in the element development.
This careful selection process ensured utility of the MultiRATE FM dataset, forming the bedrock for the forecasting methodology.
A significant challenge in developing this forecasting module was to effectively capture the intrinsic relationships among disparate features and individual data samples. The nature of the gathered data, which are unstructured, presented an ideal scenario for the application of graph-based method. The idea is that by treating the data as graphs, these inherent connections could be more powerfully exploited.
Consequently, an end-to-end graph-based method was adopted as the foundation of the forecasting module. Initially, the unstructured raw data is transformed into a fully connected graph. Following this initial construction, the forecasting module (FM) then dynamically learns and optimises the most effective graph structure with respect to the ultimate task of node classification. This adaptive approach allows the system to identify and prioritize the most salient relationships within the data, leading to more accurate and insightful predictions Marthoglou et al. (2024). This methodology moves be- yond simple correlation, enabling a deeper understanding of how different factors interdependently influence development effort.
To facilitate user interaction with the forecasting module and to present the results in an intuitive manner, a custom 3D graphical user interface (GUI) was developed. This GUI is seamlessly integrated into the MultiRATE project’s broader visual analytics dashboard, providing a unified and comprehensive user experience.
The GUI empowers users to input the specific features of the element under inspection and submit them to the forecasting service. Upon submission, the forecasting module instantly provides the estimated effort. Crucially, the inserted element is then visually represented within the GUI as a 3D object, offering a multi-dimensional perspective on the forecast. This 3D representation dynamically displays key information, including:
•
Duration Information: Numerical values and relative proportions for both the overall project and the specific element.
•
Person-Months Information: Numerical values and relative pro- portions for both the overall project and the specific element.
•
TRL Information: The TRLs of the project are represented both spatially (through the object’s dimensions) and visually (through a colour scale).
A clear legend on the right side of the interface explains the colour coding, enhancing interpretability. For detailed inspection, users can hover over the 3D object to reveal pop-up boxes containing precise data, and all element information is also accessible in an organized list at the bottom of the page (
Figure 3). Furthermore, the web application incorporates a valuable comparison feature, allowing users to juxtapose two different elements and analyse their estimated efforts side-by-side, fostering a more comprehensive understanding of development requirements (
Figure 4).
Figure 3. An example of showing one element with input fields in the left sidebar.
Figure 4. An example of elements comparison.
Evaluation with external network of experts
The presented methodology and all its components (i.e., the artefacts) have been evaluated by the MultiRATE project’s stakeholder network; a diverse group of experts and organizations, including research projects and supporting entities, that represents the EU’s security Research and Innovation (R&I) landscape. To structure the network, members were categorized by type and by their level of engagement: “hard engagement” involved active collaboration under legal agreements like a Non-Disclosure Agreement (NDA) or Memorandum of Understanding (MoU), while “soft engagement” entailed occasional feedback through surveys or events without such commitments. To ensure a representative sample, the Research & Innovation (R&I) projects within the network were further classified by thematic security area. These areas include Border Management (BM), Disaster Resilient Societies (DRS), Fighting Crime and Terrorism (FCT), Digital Security (DS), and Resilient Infrastructure (INFRA).
Table 8 and
Table 9 summarize the number of members and their distribution based on these criteria.
Table 8. MultiRATE stakeholder network summary.
| Category | Identified | Hard | Soft | Total | Eng. rate | Weight |
|---|
| Security R&I projects | 87 | 17 | 16 | 33 | 38% | 60% |
| Supporting entities | 15 | 3 | 3 | 6 | 40% | 11% |
| EC and agencies | 4 | 0 | 2 | 2 | 50% | 4% |
| Other entities and experts | 32 | 5 | 9 | 14 | 44% | 25% |
| Total | 138
| 25
| 30
| 55
| 40%
| 100%
|
Table 9. R&I projects engaged.
| Area | Identified | Hard | Soft | Total | % |
|---|
| BM | 14 | 2 | 3 | 5 | 15% |
| DRS | 21 | 3 | 2 | 5 | 15% |
| FCT | 25 | 10 | 5 | 15 | 45% |
| DS/CS | 18 | 1 | 2 | 3 | 9% |
| INFRA | 6 | 1 | 1 | 2 | 6% |
| Others | 3 | 0 | 3 | 3 | 9% |
| Total | 87
| 17
| 16
| 33
| 100%
|
Validation plan
To conduct the design and validation activities, Design Science Research Methodology (DSRM) was used, to formally implement an objective, utility based, and end-user centric approach, and conduct the activities in iterative cycles to elicit frequent feedback. DSRM incorporates principles, practices, and process models which are adequate to conduct design science research in applied research disciplines, whose cultures value incrementally effective solutions Hevner and Chatterjee (2010). The design science paradigm seeks to create and evaluate “what is effective” in the problem space Hevner et al. (2004). DSRM was preferred over alternative methodological approaches for this study. Agile prototyping approaches (Beck et al. 2001) emphasise rapid, incremental software delivery but do not provide a structured process for the rigorous evaluation of a designed artefact against defined criteria, which is central to our validation objectives. Delphi-type consensus methods (Okoli and Pawlowski 2004) are effective for eliciting expert agreement on a defined issue, but they are consensus-building tools rather than design processes; they do not prescribe how to build and iteratively refine a new artefact. In contrast, DSRM provides an explicit sequence of design and evaluation cycles in which each iteration incorporates feedback into the artefact, making it the most suitable structure for developing and validating the MultiRATE HRL (Peffers et al. 2007). For the final validation cycle, a System Usability Scale questionnaire was used, tailored to the needs of MultiRATE and mapped to DSRM.
Four design and validation cycles were scheduled to allow for frequent synchronization of insights and lessons learned. Within each cycle, regular meetings were conducted to further synchronize development, validation, and network building activities.
The evaluation activities were performed according to the following procedures:
•
Data collection: A common DSRM template was used to collect data for each MultiRATE artifacts, containing data fields for use cases, artifacts, system dimensions, evaluation criteria, objectives, formal ratings, and informal evaluators’ feedback.
•
Data analysis and reporting: Each artefact owner collected feedback received during the evaluation sessions; feedback was then shared with other artefact owners, as well as analysed and discussed during regular meetings; additionally, formal reporting was provided through project deliverables.
•
Implementation: Based on feedback received, as well as the development backlog, each artefact owner selected the set of features to be implemented for the following development iteration.
The DSRM process model adopted was based on the model developed by Peffers et al. (2007), as shown in the diagram of
Figure 5.
Validation results
In the fourth evaluation cycle, external evaluators were contacted to test the HRL web-based calculator, with all RLs now fully integrated as a part of the methodology. Overall, based on the feedback received, the HRL calculator shows potential as a useful tool but requires significant enhancements across multiple domains. A primary and widespread concern is its understandability; evaluators frequently reported that concepts were complex, wording was confusing, and results, particularly graphs, were difficult to interpret without further explanation. Users requested clearer HRL level explanations, improved guidance, and a more intuitive UI/UX design. Efficacy is another critical area needing attention, with specific feedback highlighting potentially flawed HRL calculations when Industrial or Manufacturing Readiness Levels (IRL/MRL) are not applicable or are dropped. The tool’s current primary focus on “products” rather than processes, and the absence of key assessment components such as the usability of a product by a user (Human Readiness Level) or items like the availability of training curricula, were also noted as limitations to achieving its purpose. Regarding its practical utility for users, issues such as the inability to change initial parameters once set and cumbersome UI elements like the lack of an “expand all” feature for questions were raised. For the tool to achieve better organizational fit and utility, suggestions included making it more generic beyond specific EU-funded security research actions, ensuring it encompasses a broader range of project types and funding areas, and providing clarity on its value proposition, data handling, and integration into organizational practices like proposals. Finally, a need for a greater level of detail through more sufficient contextual help and guidance within the tool was also explicitly requested by users.
Throughout the final evaluation cycle, the web-based calculator has been modified in response to the feedback received. More specifically, a couple of updates have been applied starting from additional explanations that have been added to the indicators that contained keywords that weren’t clear enough. Secondly, the wording in some explanations has been improved for use beyond EU projects or products. Then, the menu for selecting different assessments has been enhanced by including additional details such as the project name, element name, and whether it is a quick or full assessment. This improvement allows easier navigation between assessments and enables users to compare, for instance, the quick and full assessments of the same element more effectively. Additionally, the user manual has been improved, adding a step-by-step guide with a practical example showing the differences between the quick and the full assessments of an element. Last but not least, graphs have been improved with more explanations to better understand their meaning. Besides, automated feedback for improving the holistic RL of the assessed element has been added.
During the fourth evaluation cycle the System Usability Scale (SUS) questionnaire, tailored to the needs of MultiRATE, was used to simplify the evaluations and the method to receive feedback from the external participants. SUS is a well-established usability questionnaire Brooke (1996), with documented psychometric properties for post-task and post-session usability assessment Lewis (1991), which uses 10 questions, and the responses are based on the Likert scale with 5 response options. Except for the ten questions of SUS, we added one question per RL (including HRL) to measure the expectation of the evaluator compared to the results of the assessment for each RL. Six more questions were added, focused on the HRL from which two are open questions. The questionnaire was created and distributed using EU Survey, the European Commission’s online survey tool. The HRL web platform, available at https://hrl.multiratehl.iti.gr, follows a three-tier architecture (presentation, application, and data tiers) and consists of a frontend and a backend. The frontend was implemented as an Angular application (Angular 12.2.17, Bootstrap 4.5.0, Chart.js 4.4.7 for the readiness-level visualisations, Angular Material 12.2.13 for the UI components, TypeScript 4.3.5). The backend was developed with the JHipster framework (JHipster 8.1.0, Spring 3.2.0, MongoDB 4.4), which provides native Spring capabilities and integrated security mechanisms, and exposes a REST API layer that separates user management and CRUD services from the business logic (score calculation, assessment status). The investment forecasting module applies graph neural network methods (VSLII) for clustering, classifying, and visualising RL evaluation outcomes, and estimates the budget and market entry time required for a technology to reach higher maturity levels. Furthermore, MultiRATE consortium created a mapping between the SUS (
Table 10) based evaluation questionnaire and the DSRM methodology (
Table 11). The questions as well as the number of replies are depicted. For coherency purposes, the questions are presented below from i to x.
i) I think that I would like to use this system frequently.
ii) I found the system unnecessarily complex.
iii) I find the system easy to use.
iv) I think that I would need the support of a technical person to be able to use this system.
v) I found the various functions in this system were well integrated.
vi) I thought there was too much inconsistency in this system.
vii) I would imagine that most people would learn to use this system very quickly.
viii) I found the system very cumbersome to use.
ix) I felt very confident using the system.
x) I needed to learn a lot of things before I could get going with this system.
Table 10. SUS (System Usability Scale) evaluation results. The table displays response counts for ten questions (i-x).
| Response type | i | ii | iii | iv | v | vi | vii | viii | ix | x |
|---|
| Strongly Agree | 12 | 1 | 9 | 3 | 11 | 1 | 17 | 0 | 6 | 1 |
| Agree | 14 | 2 | 20 | 0 | 18 | 1 | 8 | 2 | 19 | 2 |
| Somewhat Agree | 7 | 4 | 4 | 2 | 5 | 2 | 8 | 5 | 7 | 4 |
| Somewhat Disagree | 2 | 3 | 2 | 5 | 2 | 2 | 2 | 1 | 4 | 5 |
| Disagree | 2 | 17 | 2 | 8 | 0 | 15 | 0 | 10 | 1 | 16 |
| Strongly Disagree | 0 | 6 | 0 | 11 | 0 | 8 | 0 | 9 | 0 | 5 |
| Blanks | 0
| 4 | 0 | 8 | 1 | 8 | 2 | 10 | 0 | 4 |
| Total Responses | 37
| 37
| 37
| 37
| 37
| 37
| 37
| 37
| 37
| 37
|
| Positive Responses | 89.2%
| 78.8%
| 89.2%
| 82.8%
| 94.4%
| 86.2%
| 94.3%
| 74.1%
| 86.5%
| 78.8%
|
Table 11. MultiRATE HRL evaluation criteria and objectives based on DSRM.
| System dimensions | Evaluation criteria/Subcriteria |
Objectives |
|---|
| Goal | Efficacy | The degree to which the HRL has achieved its purpose. |
| Environment | Consistency with people/Utility | While conducting the maturity evaluation of a project element, the HRL tool/method is useful for assessing it holistically. |
| Environment | Consistency with people/Understandability | While conducting the maturity evaluation of a project element, the HRL tool/method enables easy understanding of the intermediate achievements needed in order to reach the desired final achievement and comply with the project contractual obligations. |
| Environment | Consistency with organization/Utility | The HRL is useful for the services offered by the organization, it provides evidence on whether the element is on good stage. |
| Environment | Consistency with organization/Fit with organization | The HRL can be easily fit in the organization practices. |
| Structure | Level of detail | The HRL provides a sufficient level of detail needed for its goal on an executive summary level. |
Regarding the individual RLs, i.e. TRL, SocRL, SecRL, LPERL, IRL, en-CRL, and MRL, evaluators were asked to what degree the RL assessment results matched their expectations. The results demonstrated that, on average, the RL assessment results matched the evaluators’ expectations in more than 80% of the cases; we interpret this as expectation-based (perceptual) validation, reflecting the alignment between the assessment outcomes and expert expectations, rather than as predictive or empirical accuracy against an external benchmark. Additionally, regarding the evaluation on the FM, as presented in
Table 12, all 7 answers are positive, since in all answers evaluate the goals as Largely or Fully Achieved (LA and FA respectively), and the other questions are rated highly on a 1-7 Likert scale (1 answer 5/7, and all other answers 6 and above).
Table 12. Evaluation results for the forecasting module.
| Question |
Answers |
|---|
| Was the procedure clear enough for the user to follow and were the instructions adequate to understand the usage? | Yes: 5/7
Needs more tooltips/info: 1/7
Not exactly: 1/7 |
| How easy was the tool to use? (Scale 1-7) | 14.3% (5)|71.4% (6)|14.3% (7) |
| Environment Consistency with people/Utility (Scale 1-7) | 28.6% (6)|71.4% (7) |
| What is the Goal of the tool - About Efficacy | 57.1% LA|42.9% FA |
| Environment Consistency with Organization/Utility | 42.9% LA|57.1% FA |
| Structure - Level of detail | 71.4% LA|28.6% FA |
| Activity - Performance (Scale 1-7) | 57.1% (6)|42.9% (7) |
Discussion
The MultiRATE HRL framework addresses a gap identified in the related work: no existing framework concurrently evaluates the seven dimensions needed for the civil security domain, and the state of the art lacks security-specific scales such as SecRL and LPERL Adomaitis et al. (2024). Our results show that the integrated assessment, supported by the web-based calculator and the forecasting module, was evaluated positively by the external expert network in terms of usability, utility, and expectation matching. In this section we discuss how the framework relates to prior holistic approaches and reflect on its limitations.
Compared with composite-index approaches such as the DoD SRA/SRL Sauser et al. (2006), MultiRATE avoids reducing the assessment to a single opaque score. As discussed in the Method section, the four holistic levels (HRL1–HRL4) are always presented together with the full dimension-level breakdown through the spider and bar graphs, which keeps weak dimensions visible. This mitigates — although it does not fully eliminate — the critique of arithmetic operations on ordinal data Olechowski et al. (2017), because the aggregation is computed on percentages of indicator achievement within each level rather than on the ordinal level values themselves. We present this as a pragmatic trade-off consistent with the measurement properties of ordinal scales Stevens (1946) and with multi-attribute evaluation practice Keeney and Raiffa (1976). In this respect, MultiRATE follows the approach of context-specific dashboards Gerdsri and Manotungvorapun (2021); Vik et al. (2021) rather than opaque composite indices.
A further design consideration is the weighting of the dimensions in the HRL computation. In the current version, the aligned levels of the seven dimensions contribute equally to the four holistic levels, a choice made for transparency and ease of interpretation rather than as a claim that all dimensions are equally important in every context. The framework provides mechanisms for context-dependence: the initialization phase allows users to exclude non-applicable dimensions (e.g., the “Exclude LPERL” option and the “Starting xRL” concept in Table 6), and the dimension-level results remain fully visible in the dashboard, so that in cases where legal or ethical considerations dominate — such as surveillance or predictive policing technologies — these dimensions surface explicitly through the LPERL and SecRL results rather than being hidden inside a weighted composite. The exploration of configurable, context-dependent weighting schemes, potentially informed by multi-criteria decision analysis methods such as the Analytic Hierarchy Process Saaty (1980), is left for future work.
The scope of the framework spans the broad category of products, systems, and processes developed in civil security R&I projects, which includes technologies as different as low-risk administrative tools and highly sensitive systems such as predictive policing or surveillance platforms. For the latter, societal and legal dimensions may become dominant constraints rather than merely additional factors; structured ethical impact assessment has been proposed precisely for such technologies Wright (2010). The framework does not impose a fixed weighting of dimensions: the initialization phase captures the assessment context (security area, nature of the element, whether human subjects are involved, whether the element includes AI — Table 6), non-applicable dimensions can be excluded, and the results are always interpretable at the dimension level. This allows dominant constraints to surface explicitly. We recommend that, for sensitive technologies, results be interpreted primarily at the dimension level, with particular attention to the LPERL and SecRL outcomes, before considering the holistic levels.
Compared with the EU 4-Axis framework Bruno et al. (2020), which assesses digital technologies for European public services along societal, organisational, and legal axes, MultiRATE adds the commercialization and manufacturing dimensions that are vital for ensuring a technology can be produced and sustained as a viable product, as well as security-specific dimensions (SecRL and LPERL) that are critical in the civil security domain. Unlike process-focused guides such as the GAO TRA U.S. Government Accountability Office (2020), which remain centred on the TRL metric, MultiRATE operationalises a genuinely multi-dimensional assessment while retaining the evidence-based, indicator-driven discipline that process guides promote. The investment forecasting module is, to our knowledge, a distinctive feature of MultiRATE: it provides an evidence-based estimate of the effort required for maturity progression, supporting decisions on resource allocation that other readiness frameworks do not offer.
The limitations of this work are discussed in the Conclusion, and the scope of applicability across different types of civil security technologies is addressed in relation to the framework’s initialization options. Overall, the results support the MultiRATE HRL as a practical, transparent assessment tool for the civil security domain, while indicating that further refinement of usability, indicator wording, and the forecasting model is warranted, as detailed in the validation results.
Conclusion
The traditional TRL scale does not fully address the complexities associated with security innovation. This paper introduced the MultiRATE Holistic RL framework as a direct response to this challenge, providing a validated, multi-dimensional approach to maturity assessment. By harmonizing seven critical RL dimensions our framework moves beyond a narrow technical focus to offer a truly holistic view.
Our main contributions are twofold. First, we developed a unified methodology that effectively integrates various readiness scales while preserving their distinct characteristics. Second, we implemented this methodology in the publicly accessible HRL calculator, a practical tool that provides not only a comprehensive assessment dashboard but also a novel investment forecasting module to guide resource allocation.
The validation of the HRL framework through the Design Science Research Methodology confirms its utility and feasibility for the civil security domain. Ultimately, MultiRATE provides a common language for researchers, end-users, and funding bodies, helping to de-risk the innovation process and bridge the “Valley of Death.” This work serves as a model for developing similar holistic assessments in other complex, high-stakes fields, championing a more responsible and effective pathway for technological transition. Future work will focus on adapting the framework for new domains, enhancing it with additional dimensions such as a Human Readiness Level, and conducting longitudinal studies to confirm its long-term impact.
Several limitations of this study should be acknowledged. First, the validation relied on a sample of external evaluators engaged through the project’s stakeholder network (55 engaged members out of 138 identified, as summarised in Table 8), which is moderate in size and not statistically representative of the full civil security R&I community. Second, the framework was designed and validated within the civil security sector, which limits the generalisability of the results to other application domains without adaptation. Third, the validation is based on expert self-reports and expectation matching rather than on external performance benchmarks; as discussed in the validation results, the reported accuracy reflects perceptual agreement between the assessment results and the evaluators’ expectations.
The framework also offers concrete implications for policy and practice. For EU research and innovation programming, the MultiRATE HRL could complement TRL-based reporting in Horizon Europe projects by providing a more complete picture of maturity across security-relevant dimensions, supporting programme-level assessment of uptake potential. For LEAs and other end-users, the framework provides structured evidence for procurement decisions, enabling the comparison of candidate solutions across technical, societal, legal, and security dimensions. For startups and technology suppliers in the civil security market, the investment forecasting module offers an evidence-based estimate of the effort required to reach the market, which can inform funding strategies and investment roadmaps.
