Introduction
Foreign language acquisition has historically relied on structured pedagogical approaches, including grammar-translation methods, audio-lingual drills, and immersion techniques.1,2 The grammar-translation approach, rooted in classical education, emphasises textual analysis and memorisation of grammatical rules, while audio-lingual methods focus on pattern repetition and habitual formation.1 Immersion-based strategies aim to replicate natural language acquisition by embedding learners in target-language contexts.2 Over the past few decades, technological tools such as language laboratories, multimedia programmes, and computer-assisted instruction have augmented these traditional approaches, allowing learners to practice independently.2,3
The emergence of Extended Reality (XR), encompassing Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), has introduced transformative possibilities for language education by providing immersive, context-rich simulations that transcend the limitations of traditional classroom settings.3,4 AR overlays digital content onto the physical world, enhancing vocabulary acquisition through contextualised visual and auditory stimuli, while VR fully immerses users in synthetic environments where they can engage in conversational practice with virtual interlocutors.4,5 Recent studies have demonstrated that VR environments can improve learners’ speaking confidence and reduce anxiety by offering risk-free, repeatable practice opportunities.6,7 Commercial platforms such as ImmerseMe VR8 enable learners to navigate realistic scenarios, such as ordering food in a restaurant or asking for directions, thereby promoting contextual learning, while AR applications like MondlyAR9 leverage spatial anchoring to reinforce vocabulary retention.
Artificial Intelligence (AI) complements XR by powering adaptive learning systems, natural language processing (NLP), and real-time feedback mechanisms.10 AI-driven 3D avatars within XR environments can deliver personalised language instruction, facilitate conversational practice, and adapt to individual learner needs, creating a more engaging and effective educational experience.5,11 The integration of AI with XR has been identified as a key enabler of immersive, learner-centred approaches to language acquisition, closing the gap between controlled classroom exercises and authentic communicative practice.6,12
However, significant gaps remain in the accessibility and inclusivity of current XR-based language learning solutions.13,12 Most existing platforms are designed for single-language environments or narrowly defined use cases, limiting their utility for multilingual and multicultural learners.13 More critically, these systems largely neglect the needs of deaf and hard-of-hearing individuals, who require sign language support to participate effectively in language education.14,15 Although advances in AI-driven sign language recognition have been achieved, the development of comprehensive text-to-sign translation systems remains constrained by insufficient annotated datasets, low accuracy in gesture recognition, and the absence of real-time translation capabilities within XR environments.16,17 Current challenges are compounded by the diversity of sign languages across regions, with most research focusing on American Sign Language (ASL) while largely overlooking International Sign (IS), a visual communication system used by deaf individuals from diverse linguistic backgrounds in international contexts.18,19
Furthermore, while AI-driven avatars hold great promise for natural, culturally sensitive language instruction, they currently struggle to deliver accurate interactions across different languages and cultural nuances due to challenges in NLP algorithms and the complexity of implementing diverse linguistic databases.14,20 Robust pedagogical frameworks that guide the effective integration of XR technologies into language curricula are also needed to ensure that technological innovation translates into meaningful educational outcomes.12,21
The need for inclusive educational technologies is further underscored by European Union policy frameworks, including the European Accessibility Act and the European Strategy for the Rights of Persons with Disabilities 2021–2030, which call for accessible digital services and equitable participation in education across member states. The EU’s Digital Education Action Plan (2021–2027) specifically highlights the role of emerging technologies, including AI and immersive environments, in encouraging inclusive and high-quality education. Despite these policy initiatives, there remains a significant gap between the accessibility aspirations articulated in EU frameworks and the practical capabilities of current XR-based educational platforms, particularly regarding support for sign language users and multilingual learners.
This paper addresses these gaps by presenting a comprehensive platform that integrates modular AI-driven services for inclusive language learning in immersive XR environments. Building upon and extending our earlier conference publication,22,47,48 this paper provides an expanded literature review, describes the complete system architecture, presents quantitative benchmarking analyses of AI translation models and text-to-speech services, and discusses the implications for equitable educational technology. The contributions of this work can be summarised as follows: (a) the design and implementation of a modular, interoperable framework that combines speech-to-text, text-to-speech, text-to-text translation, sentiment analysis, and IS translation within a unified XR platform; (b) the creation of an IS gesture dataset comprising 750 videos processed using Google MediaPipe for real-time avatar-driven sign language delivery; (c) quantitative benchmarking of multilingual translation models (NLLB-200 vs. EuroLLM) and text-to-speech services; and (d) a scalability evaluation demonstrating robust performance under simulated high-demand scenarios.
The remainder of this paper is organised as follows. Section 2 presents the related work across XR for education, AI-driven translation, sign language processing, and session summarisation. Section 3 details the methodology and system architecture. Section 4 presents the results, including AI service implementations, and benchmarking analyses. Section 5 discusses the findings and their implications, and Section 6 concludes the paper with directions for future work.
Related work
Extended reality for language education
The application of XR technologies in education has grown substantially, driven by the recognition that immersive environments can enhance engagement, motivation, and learning retention.3,4 In the context of language learning, VR provides a unique opportunity to situate learners within authentic communicative contexts, enabling them to practice speaking, listening, and interacting in a target language without the social pressures of real-world interactions.6,7 Divekar et al.3 demonstrated that foreign language acquisition systems combining AI with XR can significantly improve learner outcomes by providing contextualised, adaptive interactions. Tegoan et al.4 conducted a systematic review of XR applications for language teaching, concluding that immersive technologies offer distinct advantages in promoting experiential learning, though they also identified limitations related to content design and pedagogical alignment.
Research by Zhi and Wu7 proposed a cognitive-affective model of immersive learning, arguing that XR-based language learning environments enhance both cognitive processing and emotional engagement, leading to deeper learning outcomes. Godwin-Jones6 explored the concepts of presence and agency in virtual spaces, highlighting the promise of XR for creating authentic language learning experiences where learners can exercise autonomy and make meaningful communicative choices. Panagiotidis5 examined VR applications specifically designed for language learning, finding that virtual environments can effectively supplement traditional instruction by offering novel modalities for practice and assessment.
Despite these advances, Taborda et al.13 noted that engagement and attention in XR learning environments remain under-researched, with limited understanding of how immersive features affect sustained learning. Zhang et al.12 identified a need for stronger theoretical frameworks to guide the integration of mixed reality technologies into language curricula, arguing that without such frameworks, the deployment of XR tools risks being technologically driven rather than pedagogically grounded. Garcia et al.10 explored the intersection of AI and XR in design education, identifying both challenges and opportunities that arise when emerging technologies are applied in educational contexts.
AI-driven speech recognition and translation in XR
Advances in automatic speech recognition (ASR) have been accelerated by deep learning architectures, particularly encoder-decoder frameworks and transformer models.23,24 OpenAI’s Whisper model represents a significant milestone in ASR, achieving robust multilingual speech recognition through training on over 680,000 hours of weakly supervised audio data.23 Whisper’s ability to generalise across languages and acoustic conditions makes it well-suited for integration into XR environments where real-time, accurate transcription is essential for inclusive communication.11,23
Multilingual translation has also seen substantial progress through models such as Meta’s No Language Left Behind (NLLB), which supports translation across 200 languages using a conditional compute architecture based on Sparsely Gated Mixture of Experts.24 The NLLB model achieves a 44% improvement in BLEU scores relative to previous state-of-the-art systems, with particular gains for low-resource languages.24 In XR contexts, real-time translation services facilitate collaboration among multilingual users, effectively overcoming language barriers and enriching user interactions.20,21 Research has focused on bridging the modality gap between speech and text to ensure effective cross-modal communication within immersive environments,25 while multilingual audio-visual corpora such as MuAViC have been developed to support robust speech-to-text translation across modalities.26
Hartholt et al.11 described a multi-platform framework for embodied AI agents in XR, demonstrating the potential of virtual humans to serve as ubiquitous interaction partners that can deliver contextualised language instruction. Sylaiou et al.20 explored the use of XR technologies for fostering visitor experience and inclusion at industrial museums, demonstrating how real-time transcription and translation can enhance accessibility in cultural settings. Hirzle et al.21 provided a scoping review of the intersection between XR and AI, mapping the landscape of research at this convergence and identifying key opportunities and challenges for future development.
Sign language translation and recognition
AI has significantly advanced sign language translation and recognition, facilitating communication between deaf and hearing individuals by converting spoken or written language into sign language gestures.17,27 Deep learning models, particularly transformer-based architectures, have led to substantial improvements in both translation accuracy and efficiency.16,28 Current systems often utilise gloss annotation, which decomposes signs into linguistic components to enhance translation quality,27 while newer gloss-free approaches have demonstrated comparable results by eliminating this intermediate step.28 Visual-language pretraining techniques have proven effective in enhancing both the accuracy and scalability of sign language translation, as demonstrated on benchmark datasets such as PHOENIX14T and CSL-Daily.27,29 Contrastive visual-textual transformation approaches, such as CVT-SLR, have been proposed for sign language recognition with variational alignment, achieving strong results on standard benchmarks.29 Additionally, modern wearable and sensor-based technologies have the potential to complement AI systems by enabling real-time gesture recognition and enhancing accessibility for individuals with hearing impairments.30 Taborri et al.14 reviewed the use of AI for sign language recognition in education, with a focus on the ISENSE project, while Strobel et al.15 applied design science research to develop AI-based sign language translation systems. Google MediaPipe has emerged as a powerful open-source framework for real-time hand and body tracking, providing 21 three-dimensional hand landmarks from single video frames.31 Its lightweight architecture and ability to run on mobile devices make it particularly suitable for integration with XR applications where computational efficiency is critical. MediaPipe has been widely adopted in sign language recognition research, enabling the extraction of precise hand and finger coordinates that serve as input features for gesture classification models.31,32 Despite these advances, significant challenges remain, particularly in developing comprehensive systems for International Sign (IS). Unlike national sign languages such as ASL or British Sign Language, IS is a visual communication system used by deaf individuals from diverse linguistic backgrounds in international contexts, such as conferences, sporting events, and educational settings.18,33 Online sign language repositories such as SpreadTheSign45 have contributed to cross-linguistic accessibility by providing video demonstrations of signs across multiple national sign languages, serving as valuable training resources for computational models. Recent work by Srivastava et al.46 demonstrated the effectiveness of MediaPipe Holistic combined with deep learning architectures for continuous sign language recognition, achieving promising results that underscore the potential of landmark-based approaches for real-time gesture classification. However, while some progress has been made with ASL translation models,34 advanced implementations for IS remain scarce, and the animation of sign language gestures via avatars in XR environments represents an open research challenge.19,22
Large language models for session summarisation
Large Language Models (LLMs) have advanced the capabilities of automatic summarisation by generating coherent and contextually appropriate summaries of complex data.35,36 Models such as GPT-4 and fine-tuned variants of T5 and BART excel at processing large datasets and extracting relevant information, making them well-suited for session summarisation in educational contexts.36 In XR environments, LLMs can synthesise immersive experiences by summarising key interactions, enabling learners to revisit essential insights from their sessions. The ability of LLMs to perform abstractive summarisation allows them to generate coherent narratives rather than mere repetitions of content, enhancing learning retention.36 Bozkir et al.37 explored the embedding of LLMs into XR, identifying opportunities and challenges for inclusion, engagement, and privacy. Boros and Oyamada36 investigated LLM organisation for abstractive summarisation, while Ramprasad et al.38 analysed LLM behaviour in dialogue summarisation, unveiling trends in circumstantial hallucinations that can affect factual accuracy. Addressing hallucination through fine-tuning and task-specific training remains an important area for solidifying the role of LLMs in adaptive learning systems within XR.38
Sentiment analysis in educational technology
Sentiment analysis has gained traction as a means of enriching communication in educational technology by detecting and conveying emotional cues. Transformer-based models such as RoBERTa have demonstrated strong performance in classifying textual inputs into emotional categories.39 In the context of XR-based language learning, sentiment analysis can enrich avatar-mediated communication by mapping detected emotions to visual cues, such as emoticons, thereby providing additional contextual information that enhances emotional clarity and social presence for learners.22,39
Methodology
Conceptual framework and system architecture
The proposed platform integrates modular AI-driven services designed for both deaf and hearing individuals within immersive XR environments. The system architecture, illustrated in
Figure 1, follows a service-oriented approach where each AI capability, speech-to-text, text-to-speech, text-to-text translation, text-to-sign translation, sentiment analysis, and session summarisation, is deployed as an independent microservice accessible via RESTful APIs. This modular design ensures flexibility, scalability, and interoperability across XR platforms.
Figure 1. High-level description of the proposed system architecture.
The services are hosted on AWS Cloud infrastructure, ensuring high availability and the ability to scale horizontally to accommodate increasing user loads. In VR settings, 3D avatars function as virtual tutors, delivering real-time language learning through text and speech translation for hearing users and text-to-sign translation for deaf users. The XR scenarios were designed in consultation with educators and language acquisition experts to ensure pedagogical relevance, aligning AI services with established communicative and task-based learning approaches.
To ensure scalability, the platform employs Docker-based orchestration and cloud-native compatibility, enabling horizontal scaling of individual AI microservices. A load testing campaign simulated 1,000 concurrent users sending simultaneous API requests to the backend services, with the system maintaining an average response time under 800 milliseconds and recording no critical failures, confirming robust performance under high-demand educational scenarios.
Speech-to-text transcription
The speech-to-text component leverages OpenAI’s Whisper model,23 an open-source ASR system trained on over 680,000 hours of multilingual and multitask supervised data. Whisper employs a sequence-to-sequence transformer architecture that processes audio spectrograms and generates transcriptions, supporting robust performance across diverse languages and acoustic conditions. The platform deploys Whisper as an API service that receives audio input from users within the VR environment and returns real-time text transcriptions.
Figure 2 illustrates the speech-to-text transcription process in Greek, demonstrating accurate conversion of spoken inputs into text. The terminal output shows real-time transcription of Greek speech input, demonstrating accurate processing of spoken language into text for subsequent translation and display within the XR environment.
Figure 2. Speech-to-text transcription using a Whisper AI wrapper.23
Multilingual text-to-text translation
For multilingual translation, the platform integrates Meta’s No Language Left Behind (NLLB) model,24 a conditional compute model based on Sparsely Gated Mixture of Experts that supports translation across 200 languages. The NLLB-200-distilled-600 M variant is deployed as a translation API service, enabling real-time conversion of transcribed text between multiple language pairs. The translation service is chained sequentially with the speech-to-text service, creating an automated pipeline where user speech is first transcribed into text and then translated into the target language selected by the user.
Figure 3 demonstrates the text-to-text translation workflow using NLLB, showcasing real-time translation across multiple languages. The system translates text in real time across multiple languages, enabling multilingual interactions within the XR learning environment.
Figure 3. Text-to-text translation workflow using Meta’s NLLB model.24
Text-to-speech synthesis
The text-to-speech (TTS) component generates natural, real-time audio output in the target language. Initially, the platform explored the Coqui TTS Tacotron2-DDC model40 and the open-source Piper TTS system.41 However, due to security vulnerabilities associated with poorly maintained Python dependencies in Piper and limitations in voice naturalness with Tacotron2-DDC, the team transitioned to AWS Polly, a production-grade TTS service that offers low-latency, high-quality speech synthesis across 34 languages.
The selection of AWS Polly was informed by a comprehensive benchmarking study comparing four TTS services: AWS Polly Standard, Google Cloud TTS Standard, Microsoft Azure Speech, and ElevenLabs.
Table 1 presents the latency performance metrics, which were derived from both the Picovoice TTS Latency Benchmark study42 and our own testing.
Table 1 provides the latency performance metrics for text-to-speech services. AWS Polly was selected for its consistently low first-byte latency (50–100 ms), cost-effectiveness ($4 per million characters), and sufficient language and voice variety for the target application. The TTS service is integrated with the translation pipeline, providing multilingual spoken feedback through the 3D avatar system.
Table 1. Latency performance metrics for text-to-speech services.
| Service | First byte latency | FTTS |
Total response time |
|---|
| AWS Polly Standard | 50-100 ms | 450 ms | 780 ms |
| Google Cloud Standard | 300-2000 ms | 600 ms | 1200 ms |
| Microsoft Azure | 150-800 ms | 1140 ms | 1500 ms |
| ElevenLabs | 300-500 ms | 840 ms | 1250 ms |
Sentiment analysis with emoticon mapping
To enrich avatar-mediated communication with emotional context, the platform incorporates a sentiment analysis module based on the twitter-roberta-base-sentiment model from CardiffNLP.39 This RoBERTa-based transformer processes translated text input and classifies it into sentiment categories such as “happy,” “neutral,” or “sad.” Each detected sentiment is mapped to a corresponding emoticon that is displayed in real time within the VR scene adjacent to the avatar, enhancing affective expressiveness and social presence.
Figure 4 demonstrates the emoticon-based sentiment feedback displayed in the XR environment. The RoBERTa-based sentiment classifier processes the avatar’s speech output and maps detected emotions to visual emoticons displayed alongside the avatar, providing additional contextual and emotional cues for learners.
Figure 5 shows the API request and response format for the sentiment analysis module, illustrating the RESTful interface through which the VR application communicates with the sentiment service. The interface accepts JSON-formatted text input and returns classified sentiment labels with associated confidence scores for multiple emotional categories.
Figure 4. Emoticon-based sentiment feedback in the XR environment.
Figure 5. API request and response format for the RoBERTA-based sentiment analysis module.
Meeting summarisation module
The platform integrates the flan-t5-base-samsum model,43 available on Hugging Face, for real-time summarisation of dialogues in multilingual XR-based educational scenarios. This model is deployed as an API on AWS Lambda as part of the platform’s backend services. It is designed for deaf or hard-of-hearing users, interpreters, and educators to receive concise, natural language summaries of verbal exchanges within the XR scene.
Figure 6 presents the API interface for the summarisation module, showing how input dialogue is processed and condensed into a coherent summary. The system processes dialogue text and generates concise summaries, with the response including both the original text and the summarised output along with token count statistics.
Figure 6. API request and response format for the meeting summarisation module based on the flan-t5-base-samsum model.43
International Sign (IS) translation
A key contribution of this work is the development of an IS translation pipeline. During the research phase, an extensive review revealed that while each country has its own distinct sign language with unique linguistic structures, IS serves as a broadly understood visual communication system used by deaf individuals from diverse linguistic backgrounds in international contexts.18,33 Unlike national sign languages, IS draws on signs from multiple sign languages, iconic gestures, and universal visual cues to enable understanding across national boundaries.
To develop the IS translation model, approximately 750 videos of IS gestures were collected and processed using Google MediaPipe31 and OpenCV,44 extracting key movement coordinates and hand position data from 21 three-dimensional hand landmarks per frame. The MediaPipe Hands solution provides a machine learning pipeline that infers hand landmarks from single frames, outputting 21 key points per hand with x, y, and z coordinates normalised to the image frame. For each gesture video, frames were extracted at a uniform sampling rate and processed through the MediaPipe pipeline to obtain temporal sequences of landmark positions. These sequences were then normalised relative to the wrist landmark to account for variations in hand size, camera distance, and signer morphology. The normalised landmark sequences were aggregated into a structured dataset associating each sequence with its corresponding IS sign label. Resources such as HandSpeak34 and SpreadTheSign45 provided reference video demonstrations of IS signs, which were used both for dataset curation and validation of sign-to-label mappings.
This dataset served as the basis for training a gesture classification model that maps text inputs to corresponding IS signs. An API was developed to map the classified hand positions and gestures to 3D avatar animations within the Unity-based VR environment, enabling real-time IS interpretation. The avatar animation system translates the classified landmark sequences into joint rotations applied to the avatar’s skeletal rig, with interpolation between keyframes to ensure smooth transitions. Preliminary evaluations show that avatar animation latency remains consistently under 300 milliseconds, ensuring natural, real-time communication for XR users.
Figure 7 presents a visual sequence of the real-time avatar animation pipeline, showing how extracted gesture landmarks are translated into avatar movements. The left panels show the original hand gesture videos with overlaid MediaPipe landmarks (in red and blue), while the right panels illustrate the corresponding 3D avatar performing the recognised sign within the VR environment.
Figure 7. Visual sequence of real-time avatar animation driven by extracted gesture landmarks from the IS dataset.
Immersive VR learning environment
The final application was built using the Unity game engine and deployed on Meta Quest 3 headsets, providing a fully immersive learning experience. The Unity development environment was chosen for its cross-platform compatibility, extensive asset ecosystem, and native support for XR development through the XR Interaction Toolkit and OpenXR standards. The Meta Quest 3 headset was selected as the target deployment platform due to its standalone operation (requiring no tethered PC), high-resolution passthrough capabilities for potential AR extensions, and growing adoption in educational and enterprise settings.
Figure 8 illustrates the immersive VR classroom environment where a 3D avatar stands in a virtual classroom equipped with multilingual AI tools. Users interact with the system by selecting their preferred language through an interactive questionnaire presented within the VR interface; in response, the avatar provides real-time translations in both spoken language and IS-based sign translations of selected words and phrases. The avatar’s animation system supports lip-sync with generated speech output and gestural animation for IS signs, with blending between idle, speaking, and signing states managed through Unity’s Animator Controller.
Figure 8. Immersive VR-based language learning system.
This environment demonstrates how speech-to-text, text-to-sign translation, and text-to-speech services converge to create equitable, self-directed learning opportunities for both deaf and hearing users. The AI-powered avatar delivers educational content in a simulated classroom environment, translating spoken language into International Sign (IS) within a multilingual VR setting. The environment is designed for equitable access and cross-linguistic interaction, with all AI services hosted on scalable AWS Cloud infrastructure.
Figure 9 shows a snapshot of the pipeline from voice input to sign and spoken output in the Meta Quest 3 environment. The avatar is shown delivering IS signs in the virtual classroom environment, with text display panels visible for transcription and translation output.
Figure 9. MVP demonstration showing multilingual avatar interaction in VR.
Results
System implementation and integration
The proposed platform successfully integrates six AI-driven services into a cohesive XR learning experience.
Table 2 summarises the implemented services, their underlying models, and key performance characteristics.
Table 2. Summary of AI Services implemented in the proposed platform.
| Service | Model/Technology |
Key characteristic |
|---|
| Speech-to-text | OpenAI Whisper23 | Multilingual ASR, 680 k + hours training data |
| Text-to-Text translation | Meta NLLB-20024 | 200 languages, Mixture of Experts architecture |
| Text-to-Speech | AWS Polly | 34 languages, Mixture of Experts architecture |
| Sentiment | RoBERTa | Multi-class emotion classification |
| Session Summarisation | flan-t5-base-samsum43 | Abstractive dialogue summarisation |
| IS Translation | MediPipe and AVATAR animation31 | 750 gesture videos, <300 ms animation latency |
TTS benchmarking
Beyond the latency analysis presented in
Table 1, voice quality metrics were also evaluated.
Table 3 presents the Mean Opinion Score (MOS) and Word Error Rate (WER) values reported by service providers for the compared TTS services.
Table 4 summarises the service capabilities, which are important considerations for ensuring the platform remains accessible. Based on the benchmarking, AWS Polly Standard was selected for the proposed platform due to its lowest and most consistent first-byte latency (50–100 ms), cost-effectiveness ($4 per million characters), and consistent performance across testing sessions, avoiding the high variability observed with other services.
Table 3. Voice quality metrics for text-to-speech services.
| Service | Mean Opinion Score (MOS) |
Word error rate |
|---|
| AWS Polly Standard | 3.5–3.8 | 4.2 |
| Google Cloud Standard | 3.2–3.5 | Variable |
| Microsoft Azure | 3.8–4.0 | 3.0 |
| ElevenLabs | 3.83–4.2 | 2.83 |
Table 4. Service capabilities of TTS services (standard models).
| Service | Languages | Voices |
Max length |
|---|
| AWS Polly Standard | 34 | 66 | 3000 |
| Google Cloud Standard | 40+ | 220+ | 5000 |
| Microsoft Azure | 140+ | 110+ | 5000 |
| ElevenLabs | 29 | 5000+ | 5000 |
Benchmarking NLLB against EuroLLM
To evaluate potential improvements to the translation component, a comprehensive benchmark comparison was conducted between the deployed NLLB-200-distilled-600 M model and two variants of the EuroLLM 1.7B model. All experiments were performed on a consumer-grade workstation equipped with an NVIDIA GeForce RTX 4060 GPU (8 GB VRAM), ensuring that the benchmarking conditions reflect realistic deployment scenarios rather than high-end server infrastructure. The benchmarking used a test dataset of 10 English-to-French translations with varying complexity levels: simple conversational phrases (3 examples), medium-complexity technical sentences (4 examples), and complex sentences with specialised terminology (3 examples). Models were loaded sequentially to manage the limited GPU memory, and inference was conducted using float16 precision to maximise throughput within the available VRAM. For each model, translation quality (BLEU scores), inference speed, and resource utilisation were measured.
Table 5 presents the comparative results.
Table 5. NLLB vs EuroLLM performance comparison.
| Metric | NLLB-200 | EuroLLM 1.7B Base |
EuroLLM 1.7B Instruct |
|---|
| Average BLEU Score | 79.25 | 27.58 | 84.34 |
| Average Translation Time (s) | 0,596 | 1.509 | 0.529 |
| Model Load Time (s) | 26.63 | 25.37 | 40.99 |
| Memory Usage (GB) | ~2.5 | ~3.5 | ~3.5 |
| Successful Translations | 10/10 | 10/10 | 10/10 |
The key findings from the benchmarking are as follows. The EuroLLM 1.7B Instruct model achieved the highest BLEU score (84.34), outperforming NLLB (79.25) by approximately 5 points, demonstrating superior translation quality for European language pairs. However, the EuroLLM Base model performed poorly (27.58), highlighting the critical importance of instruction tuning for translation tasks. In terms of inference speed, EuroLLM 1.7B Instruct demonstrated the fastest average translation time (0.529 s), marginally faster than NLLB (0.596 s), while the base model was significantly slower at 1.509 s per translation. Purpose-built sequence-to-sequence models (NLLB) showed strong performance, while causal LM models averaged 61.37 BLEU across all variants. The EuroLLM models require more memory (~3.5 GB vs. ~2.5 GB for NLLB’s distilled version), with longer initial loading times for the instruction-tuned variant. The results present a compelling case for considering EuroLLM 1.7B Instruct as an alternative to NLLB in the translation pipeline, particularly given its specialised focus on European languages that aligns with the proposed platform’s target markets.
Discussion
The results demonstrate that the proposed platform successfully integrates multiple AI-driven services into a cohesive XR learning environment that addresses both the communication needs of hearing users and the accessibility requirements of deaf users. The modular, service-oriented architecture enables independent scaling and updating of individual components, which is critical for maintaining system performance as user demands grow and AI models evolve.
The successful integration of six AI services, speech-to-text, text-to-text translation, text-to-speech, sentiment analysis, session summarisation, and IS translation, within a unified XR environment validates the premise that combining multiple AI modalities within immersive settings can create meaningful, accessible learning experiences. The IS-focused approach, leveraging International Sign Language as a lingua franca rather than a single national sign language, has the potential to broaden accessibility beyond national sign language boundaries and serve deaf learners from diverse linguistic backgrounds.
The avatar-mediated sign language delivery, while achieving animation latency under 300 milliseconds, represents an initial proof of concept that would benefit from further refinement. In particular, the quality of non-manual markers (facial expressions, head movements, body posture) is recognised in the broader literature as essential for comprehensible and natural signing,16,19 and their integration into the avatar system represents an important area for future development.
The benchmarking analyses provide quantitative evidence for technology selection decisions. The TTS evaluation confirmed that AWS Polly offers the best balance of latency, cost, and consistency for real-time VR applications, while the NLLB vs. EuroLLM comparison revealed that instruction-tuned causal language models can outperform purpose-built translation models in both quality and speed for European language pairs. This finding has implications not only for the proposed platform but also for the broader UTTER project’s development of European language models.
Several limitations should be acknowledged. The IS dataset of 750 gesture videos, while sufficient for an initial proof-of-concept, represents a limited vocabulary that constrains the expressiveness of the translation system. Additionally, formal usability studies with standardised instruments (e.g., System Usability Scale, NASA-TLX for cognitive load) have not yet been conducted and represent an important direction for future validation. The system has not yet been evaluated with end users wearing VR headsets in naturalistic learning scenarios, which is necessary to assess immersion, spatial presence, and actual learning outcomes.
The ethical dimensions of the platform were addressed through adherence to GDPR regulations, anonymisation of voice and gesture data, secure API access controls, and the application of FAIR data principles. Efforts were made to ensure fairness across languages and cultural contexts, particularly in the design and training of sign language avatars, to avoid misrepresentation and bias.
The platform presented in this study addresses a key gap identified in the literature: the absence of integrated, multimodal AI systems within XR that serve both deaf and hearing learners simultaneously. While previous work has typically focused on individual AI capabilities in isolation, such as speech recognition for captioning23 or sign language recognition for classification,16,46 our system demonstrates the feasibility of orchestrating multiple AI services into a unified educational experience. This approach aligns with the vision articulated by Hirzle et al.,21 who identified the convergence of XR and AI as a high-potential research frontier, and extends it by grounding the integration in a concrete, deployable platform validated through comprehensive technical benchmarking.
From a policy perspective, the platform’s focus on IS as a lingua franca for deaf communication across national boundaries aligns with the objectives of the European Accessibility Act and the EU’s commitment to digital inclusion. By providing an XR-based learning environment that supports both spoken multilingual interaction and sign language translation, the proposed platform contributes to the broader agenda of equitable access to education, as articulated in the EU Digital Education Action Plan (2021–2027). The modular, cloud-native architecture ensures that the platform can be adapted to diverse institutional contexts, from formal language schools to informal community-based learning settings. Furthermore, the same underlying AI services have demonstrated applicability beyond language education, including accessible communication in professional and business meeting contexts.47
Conclusions
This paper presented a comprehensive framework for integrating modular AI-driven services into immersive XR environments to support language learning for both deaf and hearing individuals. The platform combines speech-to-text transcription, multilingual translation, text-to-speech synthesis, sentiment analysis, and International Sign translation, all delivered through AI-powered 3D avatars within a Unity-based VR environment deployed on Meta Quest 3 headsets.
The system’s modular architecture and successful integration of all AI components demonstrate the technical feasibility of the approach. Quantitative benchmarking of TTS services and multilingual translation models provided evidence-based justification for technology selection and identified promising alternatives for future integration.
Future work will focus on several priorities. First, the IS vocabulary will be significantly expanded by collecting and processing additional gesture videos from diverse interpreters, with improved gesture landmark normalisation to ensure smoother and more consistent avatar animations across signers. Second, avatar facial expressions and lip-sync functionality will be developed to provide the non-manual markers that are essential for natural sign language communication. Third, the single-user experience will be extended into a multiplayer VR setting where hearing and deaf users can communicate in real time, better showcasing the integrated sentiment analysis and translation capabilities and enabling collaborative language learning scenarios. Fourth, formal user studies with larger and more diverse participant groups will be designed and conducted, incorporating standardised usability instruments (e.g., System Usability Scale, NASA-TLX), pre/post learning assessments, and cognitive load measurements to produce quantitative evidence of learning effectiveness. Fifth, the EuroLLM 1.7B Instruct model will be further evaluated for integration into the production translation pipeline, with expanded benchmarking across additional European language pairs and domain-specific terminology. Sixth, the platform architecture will be extended to support additional XR hardware beyond Meta Quest 3, including standalone AR devices and desktop-based VR systems, to maximise accessibility across institutional contexts. Finally, the platform will be piloted in formal educational settings, including schools and language training centres, to assess real-world adoption, learning outcomes, and alignment with the EU Digital Education Action Plan’s goals for inclusive, technology-enhanced learning.
Declaration of generative AI and AI-assisted technologies in the writing process
During the preparation of this manuscript, the authors used generative AI-based tools, specifically large language models such as ChatGPT (OpenAI), solely to assist with language editing and consistency checks in limited sections. All substantive decisions regarding the design of the review, screening, data extraction, analysis and interpretation were made by the authors. The authors carefully reviewed and edited all AI-assisted text and take full responsibility for the integrity and accuracy of the manuscript’s content.
Open access software and AI models
All core AI models implementations central to the novelty and findings of this research utilise open-source libraries available under permissive licences for reuse:
These repositories contain the complete source code, model configurations, and training procedures necessary to replicate the AI integration methodology described in this paper.
Ethics and consent statement
This research did not involve direct experimentation with human participants. The International Sign (IS) gesture dataset used in this study was curated from publicly available, open-source video resources, specifically HandSpeak and SpreadTheSign, which are freely accessible online for research and academic purposes under permissive licences. No new video recordings of human signers were produced as part of this work. The gesture videos were processed computationally using Google MediaPipe to extract anonymised hand landmark coordinates; no personally identifiable information was retained in the resulting dataset. The AI models employed in the platform (Whisper, NLLB, RoBERTa, flan-t5-base-samsum) are publicly available open-source models used in their original or fine-tuned forms, and no additional training on personal data was conducted. The system architecture and all API services were designed with privacy-by-design principles, processing user inputs transiently without storing personally identifiable speech, text, or gesture data, in compliance with the European Union General Data Protection Regulation (GDPR). No adverse events were reported during the development and testing of the platform. As no human participants were involved in any stage of this research, ethical approval and informed consent were not required.
