1. Introduction
Artificial intelligence (AI) is advancing at an unprecedented pace and is increasingly positioned as a transformative force across multiple sectors of contemporary society, including education (Adiguzel et al., 2023). Recent developments in Machine Learning (ML), Natural Language Processing (NLP), and generative AI have enabled the creation of technologies capable of supporting learning, teaching, assessment, and educational decision-making in increasingly sophisticated ways (Holmes & Tuomi, 2022). Large-scale reviews of AI use in K–12 and higher education contexts consistently highlight its potential to personalize learning, provide timely feedback, and foster more adaptive and responsive educational environments (Chen et al., 2020a; Zafari et al., 2022). As a result, AI is widely portrayed as a key driver of educational innovation, capable of addressing long-standing challenges related to learner diversity, instructional efficiency, and student support (Kamalov et al., 2023).
Over the past decade, AI-powered educational tools have evolved from relatively narrow adaptive systems, such as automated assessment platforms, towards more interactive and conversational applications, including Intelligent Tutoring Systems (ITS) and conversational agents that engage learners through dialogue, explanation, and guided inquiry (Fernández-Herrero, 2024; Labadze et al., 2023; Pérez et al., 2020). This evolution has shifted research attention away from purely technical performance towards questions of pedagogical value and educational impact (Chiu et al., 2023). In particular, scholars increasingly examine how AI tools may influence students’ learning experiences by supporting self-regulation, fostering active engagement, and enabling more individualized learning trajectories (Ayeni et al., 2024; Heeg & Avraamidou, 2023; Jia et al., 2024).
Empirical research suggests that AI-supported learning environments can influence students’ understanding of subject content (Alneyadi & Wardat, 2023), motivation, and engagement with learning tasks (Badarudin et al., 2024), and overall learning experiences (Lee et al., 2022). Conversational agents and chatbot-based systems, in particular, have been associated with increased learner participation, perceived support, and motivation, as they offer accessible, non-judgmental interaction and immediate responses to student input (Kuhail et al., 2023). However, the magnitude and consistency of these effects vary considerably across studies, with outcomes strongly shaped by instructional design, duration of use, and the role of the teacher in mediating AI-supported activities.
Despite growing interest and experimentation with AI tools among students and educators, their systematic integration into everyday school practice remains limited. Multiple reviews point to a persistent gap between technological innovation and pedagogical implementation (Alasadi & Baiz, 2023; Chen et al., 2020b), noting that many AI tools are introduced without sufficient alignment with curriculum goals, learning theory, or classroom realities (Kavitha & Joshith, 2024). Moreover, a substantial proportion of existing studies are conducted in controlled or short-term experimental settings, offering limited insight into how AI functions under authentic classroom conditions characterized by time constraints, curricular pressures, and heterogeneous learner needs.
Three interrelated gaps in the literature are particularly salient. First, despite the growing availability of generative AI technologies, there remains limited empirical evidence on the educational impact of AI applications that are specifically designed for pedagogical purposes and aligned with curricular and instructional objectives. Much existing research focuses on general-purpose platforms that were not originally developed for learning contexts, making it difficult to draw robust conclusions about their educational value. Second, many studies continue to rely on controlled or short-term implementations, providing insufficient evidence on how AI-supported tools operate within everyday classroom practices and established instructional routines. Third, there is a notable lack of comparative, cross-national research examining how AI-supported learning tools function across different educational systems and institutional contexts. This constrains the generalizability of existing findings and limits understanding of how contextual factors shape educational outcomes.
In response to these gaps, the present study investigates the educational impact of Study Buddy (SB), a purpose-built AI study assistant designed to support students’ learning activities. Unlike general-purpose AI platforms, SB was specifically developed for educational use and aligned with pedagogical objectives. The study examines its integration within regular classroom instruction, thereby capturing patterns of use and learning outcomes under authentic school conditions. Furthermore, by adopting a shared framework, involving secondary schools in Cyprus and Greece, the study offers a comparative perspective on the contextual stability of observed effects.
Guided by this framework, the study seeks to contribute empirically grounded evidence on how purpose-built AI-supported learning tools relate to key aspects of students’ learning experiences. Specifically, it addresses the following research questions:
RQ1: To what extent does the use of an AI-supported study assistant influence students’ academic performance in secondary education?
RQ2: To what extent does the use of an AI-supported study assistant influence students’ motivation and engagement in learning activities?
2. Methodology
This study was conducted within the framework of the AI4EDU: Conversational AI Assistant for Teaching and Learning project (AI4EDU, project ID: 101087451, ERASMUS-EDU-2022-PI-FORWARD), a European initiative aimed at the design, development, and empirical evaluation of two conversational AI applications for school education: Study Buddy (SB), a student-facing study assistant, and Teacher Mate (TM), a teacher-facing support platform. The overarching goal of AI4EDU is to investigate how these AI-based applications can support teaching and learning processes under authentic classroom conditions and to generate evidence-informed guidance for their pedagogical integration.
The study adopts a focused, cross-national approach and reports findings from two school-based case studies implemented in Cyprus and Greece, both of which followed a shared evaluation framework and employed comparable instruments, enabling meaningful cross-case comparison. The present paper concentrates exclusively on the student-facing application, and examines its impact on students’ academic performance, motivation, and engagement.
2.1 Study Buddy: An AI conversational study assistant
SB is an AI-driven application designed to function as a virtual study companion that supports students during independent and guided learning activities. Its primary purpose is to assist learners in reinforcing subject knowledge, organizing learning tasks, and developing effective study strategies through natural language interaction.
SB offers a broad set of pedagogically oriented tools, including:
•
Summarizer: Condenses lengthy texts into concise summaries, enabling students to focus on key information.
•
Interactive concept exploration: Facilitates dynamic, guided sessions to deepen students’ understanding of complex topics.
•
Chronological order of events: Generates detailed timelines from inputted events or time periods, complete with explanatory transitions and comprehension questions.
•
Extract key points: Identifies critical elements in texts, making notetaking and study preparation more efficient.
•
Revise tool: Reviews student-written assignments, offering feedback and actionable suggestions for improvement.
•
Auto-grade exercise: Evaluates responses to questions instantly, providing detailed feedback to support continuous learning.
•
Explain term: Delivers explanations of academic terms and concepts, helping students grasp essential knowledge.
•
Text adaptation: Modifies text to align with students’ reading abilities, skills, and learning objectives.
• Conversations across time: Encourages critical thinking and creativity by allowing students to engage in imaginative dialogues with historical figures, cultural icons, or scientists.
•
Debate tool: Prepares students for debates by simulating an interactive experience where the AI acts as a debate partner.
•
Self-assessment quiz: Creates customized quizzes based on students’ study materials, promoting mastery of topics through self-assessment.
The application also includes a dashboard environment where students can complete teacher-assigned tests, review feedback, and track learning progress over time. Interaction with SB can take place through text- or voice-based input, allowing flexible and intuitive use. Overall, SB is intended to support self-directed learning while remaining adaptable to teacher guidance and curricular goals.
2.2 Evaluation design and implementation procedures
A quasi-experimental design was employed in both case studies. In each participating school, intact classes were assigned to either a control group or an experimental group, following the school’s existing classroom structure. Group assignment was conducted by the participating teachers, with attention to maintaining comparable group sizes and broadly similar prior academic profiles.
Prior to the classroom implementation, preparatory training sessions were conducted by the research team with the participating teachers. These sessions aimed to introduce foundational AI concepts, address pedagogical and ethical considerations related to AI use in education, and familiarise teachers with the functionality and intended educational use of both SB and TM.
In both cases, the instructional intervention focused on the teaching of a specific curriculum-aligned topic over a defined instructional period corresponding to the time normally allocated to that unit. All students received regular classroom instruction delivered by their teacher, following national curriculum requirements. The distinguishing factor between the two conditions concerned students’ access to SB.
Students in the experimental groups were given access to SB and participated in a familiarization session led by their teacher prior to the intervention. This session introduced the core functions of the application and outlined expectations for appropriate and purposeful use. During the implementation period, experimental-group students were encouraged to use SB to support their learning. Depending on the case, this use occurred either during independent study outside school or during structured classroom activities. In some instances, teachers provided guiding tasks to scaffold students’ interaction with the application.
Students in control groups studied the same content using conventional learning resources, such as textbooks, worksheets, notes, and teacher-provided materials, without access to SB. The intervention in both cases lasted approximately three weeks, reflecting typical instructional practice and minimizing disruption to established teaching routines.
2.3 Case study contexts
2.3.1 Case study 1: Cyprus – Physics (Grade 10)
The first case study was conducted in a public upper-secondary school in Cyprus and involved two Grade 10 Physics classes (N = 47). One class served as the control group (n = 24), while the other served as the experimental group (n = 23). The instructional unit focused on the topic of gravitational force.
This case followed a pre-test–post-test design. Prior to the instructional unit, both groups completed a curriculum-aligned academic performance test and the Motivation and Engagement Survey (MES). During the intervention, experimental-group students used SB independently outside school to review and consolidate their understanding of the topic, while control-group students relied exclusively on traditional study methods. Following the three-week intervention, the same academic test and the MES were re-administered. This design enabled examination of changes in academic performance, motivation, and engagement over time, as well as comparisons between experimental and control conditions.
2.3.2 Case study 2: Greece – History (Grade 7)
The second case study was conducted in a lower-secondary school in Greece and involved Grade 7 History classes studying an instructional unit on Byzantine emperors. The participating teacher implemented the intervention across three intact classes: one control group (25 students) and two experimental groups (19 and 26 students, respectively). For analytical purposes, the two experimental classes were treated as a single experimental condition.
The implementation lasted approximately three weeks. Students in the experimental groups used SB primarily during scheduled sessions in the school’s computer laboratory to search for and explore information related to Byzantine emperors, with emphasis on governance strategies and historical decision-making. The teacher explicitly encouraged students to critically evaluate AI-generated responses and to cross-reference information with textbooks and other authoritative sources. Students synthesized their findings and presented them in class, supporting the development of historical thinking and digital literacy.
The control group studied the same content using traditional instructional materials without access to SB. In this case, academic performance was assessed using a post-test only, administered at the end of the instructional unit. Students in both groups completed the MES before and after the intervention.
2.4 Evaluation instruments
2.4.1 Academic performance tests
In both case studies, students’ academic performance was assessed using curriculum-aligned tests aligned with the learning objectives of the respective instructional units. The tests reflected the learning objectives of the respective instructional units and corresponded to assessment practices typically employed in each subject area.
In the Cyprus Physics case, academic performance was measured using a test scored on a 100-point scale, administered both before and after the intervention. The test was based on the Newtonian Gravity Concept Inventory (NGCI) (Williamson, Willoughby, & Prather, 2013), which the participating teacher translated and adapted for use with the specific class context. It consisted of 25 multiple-choice questions on gravitational force–related content requiring justification of the selected answer.
In the Greece History case, academic performance was assessed using a paper-based post-test scored on a 20-point scale. The test consisted of two parts: (a) five true/false questions with justification and (b) five multiple-choice questions with justification. In both cases, identical assessment instruments were used for control and experimental groups.
2.4.2 Motivation and engagement survey (MES)
Students’ motivation and engagement were assessed using the Motivation and Engagement Survey (MES) developed by Lee and Reeve (2012). The MES is a validated instrument consisting of 38 items rated on a 5-point Likert scale (1 = Strongly Disagree to 5 = Strongly Agree).
The survey captures multiple dimensions of students’ learning-related motivation and engagement, including psychological need satisfaction (items 1–10), self-efficacy (items 11–15), mastery goals (items 16–18), behavioral engagement (items 19–23), emotional engagement (items 24–28), cognitive engagement (items 29–33), and agentic engagement (items 34–38).
Negatively worded items were reverse-coded so that higher scores consistently reflected more positive motivational and engagement profiles. The MES was administered to both experimental and control groups before and after the intervention in both case studies.
Responses were anonymized prior to analysis. Motivation and engagement were analyzed at the individual level and reported as group mean scores at each measurement point to examine overall trends over time while ensuring protection of student privacy.
2.5 Study Buddy usage data and implementation monitoring
To document the implementation of the intervention and to support interpretation of the study’s outcome findings, system-generated usage data from SB were collected and analyzed. The primary purpose of this analysis was to monitor implementation fidelity and student exposure to the AI application under authentic classroom conditions.
SB automatically records anonymized interaction logs at the session level for all user interactions. Each log entry includes a pseudonymized user identifier, session duration (in seconds), number of conversational turns, the conversation title and content, and the specific pedagogical tool activated during the interaction, where applicable. No personally identifiable information was collected or stored.
Usage data were collected for the full duration of the instructional intervention and were restricted to students assigned to the experimental groups. Although the interaction content was available to the research team, it was used solely to support contextual understanding of tool use and interaction patterns and was not subjected to systematic qualitative coding or discourse analysis.
Consistent with the exploratory and implementation-focused role of the usage data, analyses were conducted at a descriptive level. Usage indicators were not statistically linked to individual students’ academic performance, motivation, or engagement scores. This methodological decision was taken to avoid overinterpretation of log data and to maintain a clear distinction between implementation monitoring and outcome evaluation.
2.6 Ethical considerations, data management, and analysis
The study was conducted in accordance with national and institutional ethical guidelines for educational research Ethical approval was obtained prior to the implementation of the study in both participating countries.1
Written informed consent was obtained from both students and their parents or legal guardians prior to participation. Participants and their guardians received detailed information sheets explaining the purpose of the study, the nature of the activities involved, the voluntary nature of participation, and the procedures for data handling and privacy protection. Students and parents or legal guardians also provided their assent by signing a corresponding consent form. Participation was voluntary, and students were informed that they could withdraw from the study at any time without academic or other consequences.
All questionnaire and assessment data were anonymized prior to analysis. Individual identifiers were removed, and results were analyzed using coded datasets to ensure confidentiality. Incomplete questionnaires and missing responses were excluded from statistical analyses; therefore, sample sizes in the motivation and engagement analyses reflect only valid responses at both measurement points.
All statistical analyses were conducted using IBM SPSS Statistics for Windows, Version 29.0 (IBM Corp., Armonk, NY, USA). Descriptive statistics were computed to summarize students’ academic performance, motivation, and engagement scores. Prior to inferential analyses, the Shapiro–Wilk test was employed to assess the normality of score distributions. Depending on the distributional properties of the data, either parametric (independent-samples t-tests) or non-parametric (Mann–Whitney U tests) procedures were applied to compare experimental and control groups. Statistical significance was evaluated at the 0.05 level.
3. Results
3.1 Impact of Study Buddy on academic performance
This subsection reports findings related to students’ academic performance in the Physics and History case studies. Results are presented separately for each case due to differences in subject matter, assessment instruments, and study design.
3.1.1 Cyprus case study: Physics (Grade 10)
Students’ academic performance in the Cyprus case study was assessed using curriculum-aligned pre-test and post-test scores. Mean gain scores were calculated for each group by subtracting pre-test scores from post-test scores and were used as the primary indicator of academic progress. Descriptive statistics for pre-test scores, post-test scores, and gain scores are presented in
Table 1.
Table 1. Academic performance scores in case study 1 (Cyprus).
| Group | Size | Pre-test mean score/100 | Post-test mean score/100 | Mean gain score/100 |
|---|
| Control | 24 | 37.00 | 59.17 | 22.17 |
| Experimental | 23 | 40.87 | 71.83 | 30.96 |
Both groups demonstrated improvement from pre-test to post-test. The experimental group showed a higher mean gain score than the control group. To examine whether this difference was statistically significant, the Shapiro-Wilk test was first conducted to assess the normality of gain score distributions. As shown in
Table 2, gain scores in both groups were normally distributed (p > 0.05), supporting the use of parametric statistical analysis.
Table 2. Shapiro-Wilk test of normality for gain scores (Case study 1).
| Group | Statistic | Degrees of freedom (df
) |
Significance |
|---|
| Control | 0.979 | 24 | 0.875 |
| Experimental | 0.935 | 23 | 0.140 |
An independent samples t-test was then conducted to compare mean gain scores between the two groups. As shown in
Table 3, the difference was statistically significant at the 0.05 level, indicating higher learning gains for students in the experimental group (t = −3.56, p < 0.001). The magnitude of this difference was large, as indicated by effect size estimates: Cohen’s d = −1.044, reflecting substantially greater gain scores for the experimental group compared to the control group.
Table 3. Independent samples t-test results for gain scores (Case study 1).
| Test | t-value
| Degrees of freedom (df
) | Significance | Effect size (Cohen’s d) |
|---|
| Independent Samples t-test | −3.56 | 45 | <0.001 | −1.044 |
3.1.2 Greece case study: History (Grade 7)
In the Greek case study, students’ academic performance was assessed using a curriculum-aligned post-test administered at the end of the instructional unit. Mean post-test scores were calculated for the control group and the combined experimental group. Descriptive statistics are presented in
Table 4.
Table 4. Academic performance scores in case study 2 (Greece).
| Group | Size |
Post-test mean score/20 |
|---|
| Control | 25 | 14.28 |
| Experimental | 45 | 17.09 |
Students in the experimental groups achieved higher mean post-test scores than students in the control group. Prior to group comparison, the Shapiro-Wilk test was conducted to assess the normality of score distributions. As shown in
Table 5, post-test scores in both groups deviated significantly from normality (p < 0.05).
Table 5. Shapiro-Wilk test of normality (case study 2).
| Group | Statistic | Degrees of freedom (df
) |
Significance |
|---|
| Control | 0.905 | 25 | 0.023 |
| Experimental | 0.878 | 45 | <0.001 |
Given the violation of normality assumptions, a non-parametric Mann–Whitney U test was conducted to compare post-test scores between the control and experimental groups, as shown in
Table 6. The results revealed a statistically significant difference in academic performance, with students in the experimental group outperforming those in the control group (U = 869.00, p < 0.001).
Table 6. Mann–Whitney U test results (case study 2).
| Test | Mann-Whitney U value | Z | Significance | Effect size (r) |
|---|
| Mann-Whitney U test | 869.00 | 3.823 | <0.001 | 0.46 |
To estimate the magnitude of this difference, an effect size was calculated using the standardized test statistic (z = 3.823) and total sample size (N = 70). The resulting effect size (r = 0.46) indicates a medium-to-large effect, suggesting that the observed performance difference was not only statistically significant but also educationally meaningful.
The test results revealed a statistically significant difference in post-test performance between the experimental and control groups, with students who used SB outperforming their peers.
3.2 Impact of Study Buddy on motivation and engagement
This subsection reports results related to students’ motivation and engagement as measured by the MES. In line with the shared evaluation framework adopted across case studies, results are presented by construct (motivation and engagement) rather than by country. For both constructs, pre- and post-intervention mean scores were calculated for control and experimental groups. Mean gain scores (post-test minus pre-test) were then computed and used as the primary indicators of change over time. Prior to inferential analysis, tests of normality were conducted to determine the appropriate statistical procedures for each case.
3.2.1 Motivation
Students’ motivation was assessed using the motivation subscale of the MES (Items 1–18).
Table 7 presents pre-test and post-test mean motivation scores, along with the corresponding mean gain scores, for control and experimental groups in both case studies.
Table 7. Motivation pre-test, post-test, and mean gain scores.
| Case study | Group | Size | Pre-test mean | Post-test mean |
Mean gain |
|---|
| Physics (Cyprus) | Control | 24 | 3.60 | 3.53 | −0.07 |
| Experimental | 21 | 3.48 | 3.47 | −0.02 |
| History (Greece) | Control | 14 | 3.53 | 3.25 | −0.27 |
| Experimental | 14 | 3.58 | 3.42 | −0.16 |
Across both case studies, changes in students’ motivation over time were modest. In all groups, mean gain scores were negative, indicating slight decreases in motivation from pre- to post-test, with no substantial differences observed between experimental and control conditions.
To determine the appropriate inferential statistical tests, Shapiro-Wilk tests were conducted on motivation gain score distributions for control and experimental groups in each case study. The results are presented in
Table 8.
Table 8. Shapiro-Wilk test of normality for motivation mean gain scores.
| Case study | Group | Statistic | Degrees of freedom (df
) |
Significance |
|---|
| Physics (Cyprus) | Control | 0.910 | 24 | 0.036 |
| Experimental | 0.886 | 21 | 0.019 |
| History (Greece) | Control | 0.940 | 14 | 0.416 |
| Experimental | 0.954 | 14 | 0.621 |
The results indicated that the assumption of normality was violated for both groups in the Cyprus case (p < 0.05), whereas normality was not violated for either group in the Greece case (p > 0.05). Accordingly, non-parametric statistical testing was employed for the Cyprus case study, and parametric statistical testing was employed for the Greece case study.
For the Cyprus case study, a Mann–Whitney U test was conducted to examine differences in motivation gain scores between control and experimental groups. For the Greece case study, an independent-samples t-test was conducted to compare motivation gain scores between the two groups. The results are presented in
Table 9.
Table 9. Statistical comparison of motivation gain scores.
| Case study | Test | Test statistic | Significance |
|---|
| Physics (Cyprus) | Mann-Whitney U test | U = 245.5 | 0.882 |
| History (Greece) | Independent Samples t-test | t(28) = −0.640 | 0.528 |
The analyses indicated that differences in motivation gain scores between experimental and control groups were not statistically significant in either case study at the 0.05 level.
3.2.2 Engagement
Students’ engagement was assessed using the engagement subscale of the MES (Items 19–38).
Table 10 presents pre-test and post-test mean engagement scores, together with the corresponding mean gain scores, for control and experimental groups in both case studies.
Table 10. Engagement pre-test, post-test, and mean gain scores.
| Case study | Group | Size | Pre-test mean | Post-test mean | Mean gain |
|---|
| Physics (Cyprus) | Control | 24 | 3.34 | 3.40 | 0.06 |
| Experimental | 21 | 3.15 | 3.31 | 0.16 |
| History (Greece) | Control | 14 | 3.38 | 3.02 | −0.36 |
| Experimental | 14 | 3.35 | 3.22 | −0.13 |
Overall, changes in students’ engagement over time were limited. In the Cyprus case, both groups showed small increases in engagement, with slightly higher gains observed in the experimental group. In contrast, in the Greece case, engagement scores decreased from pre- to post-test in both groups, with larger declines observed in the control group.
To determine the appropriate inferential statistical tests, Shapiro-Wilk tests were conducted on engagement gain score distributions for control and experimental groups in each case study. The results are presented in
Table 11.
Table 11. Shapiro-Wilk test of normality for engagement mean gain scores.
| Case study | Group | Statistic | Degrees of freedom (df
) | Significance |
|---|
| Physics (Cyprus) | Control | 0.960 | 24 | 0.430 |
| Experimental | 0.944 | 21 | 0.266 |
| History (Greece) | Control | 0.915 | 14 | 0.189 |
| Experimental | 0.947 | 14 | 0.522 |
The results indicated that the assumption of normality was not violated for engagement gain scores in either group in both case studies (p > 0.05). Accordingly, parametric statistical testing was employed in both cases. Independent-samples t-tests were conducted to compare engagement gain scores between control and experimental groups in the Cyprus and Greece case studies. The results are presented in
Table 12.
Table 12. Statistical comparison of engagement gain scores.
| Case study | Test | Test statistic | Significance |
|---|
| Physics (Cyprus) | Independent Samples t-test | t = −0,975 | 0,167 |
| History (Greece) | Independent Samples t-test | t = −1,434 | 0,163 |
The analyses indicated that differences in engagement gain scores between experimental and control groups were not statistically significant in either case study at the 0.05 level.
3.3 Analysis of Study Buddy usage logs
This subsection presents descriptive results from the analysis of SB usage logs for the experimental groups in both case studies. The analysis focuses on student access to the application, intensity of use as reflected in session duration and conversational turns, and the pedagogical tools activated during interactions.
In the Cyprus Physics case, 20 of the 23 students assigned to the experimental group accessed SB at least once during the three-week instructional period, corresponding to an adoption rate of approximately 87%. In the Greece History case, usage logs indicate that 27 of the 45 students in the experimental condition accessed SB during the intervention, yielding an adoption rate of approximately 60%. Taken together, these figures indicate that SB was used by a substantial proportion of students in both contexts, although uptake was higher in the Cyprus case.
Across both case studies, students’ interactions with SB exhibited considerable variability in terms of session duration and number of conversational turns. In the Cyprus case, usage logs show a combination of brief and extended interaction sessions. Some students engaged in short sessions involving one or two conversational turns, while others interacted with the application repeatedly across the intervention period, including sessions of extended duration. This pattern is consistent with the primarily independent, out-of-school use of SB in this context and reflects differences in students’ self-directed study behaviors.
In the Greece case, interaction sessions were often longer and, in several instances, substantially extended in duration. Multiple sessions involved sustained interaction and higher numbers of conversational turns, indicating prolonged engagement with the application. This pattern reflects the structured, classroom-based implementation of SB, where students used the application during scheduled computer-lab sessions to explore historical content, elaborate ideas, and refine understanding under teacher guidance.
Analysis of tool usage patterns is aligned with the instructional focus of each case study. In the Cyprus Physics case, the most frequently used pedagogical tool was “Explain term”, suggesting that students primarily engaged with SB to clarify concepts and reinforce subject-specific understanding. Students also used a range of additional tools, including “Interactive concept exploration”, “Conversation across time”, “Debate tool”, “Revision tool”, “Extract key points”, “Text adaptation”, and “Self-assessment quiz”. A notable number of interaction sessions were logged without an explicitly selected tool, indicating free-form conversational exchanges aimed at clarification or follow-up questions (“Chat with SB” functionality).
In the Greece History case, the dominant mode of interaction was the “Conversation across time” tool, which was used extensively and often across multiple sessions. Other tools used included “Chronological order of events”, “Extract key points”, “Explain term”, “Auto-grade exercise”, “Summarizer”, and “Self-assessment quiz”. As in the Cyprus case, some interactions relied on the open-ended dialogue with the AI assistant.
Overall, the usage logs indicate that SB was actively used in both experimental contexts, with differences in adoption levels, interaction intensity, and tool-use patterns reflecting the instructional planning and pedagogical objectives of each case study.
4. Discussion
4.1 Summary of findings
This study examined the educational impact of a purpose-built AI conversational study assistant on students’ academic performance, motivation, and engagement in two secondary school contexts in Cyprus and Greece. The findings indicate that the use of SB was associated with statistically significant improvements in academic performance in both case studies, whereas no statistically significant effects were observed for students’ motivation and engagement.
In both contexts, usage log analysis confirmed that students in experimental groups engaged with the application during the intervention period. Although adoption and interaction patterns differed across cases, the logs indicate that SB was used in ways consistent with the instructional design and pedagogical aims of each setting.
4.2 Interpretation of academic performance outcomes
The positive association between SB use and academic performance aligns with prior research highlighting the potential of AI-supported learning environments to enhance students’ understanding and practice of curricular content (Alneyadi & Wardat, 2023; Dahri et al., 2024; Lee et al., 2022). Features such as interactive explanations, structured prompts, and opportunities for self-assessment likely supported retrieval practice, elaboration, and metacognitive monitoring.
The usage analytics provide important contextual insight into these outcomes. In the Cyprus Physics case, high adoption, and predominant use of concept-focused tools, particularly “Explain term”, suggest that SB functioned mainly as a revision and clarification aid during independent study. In the Greece History case, students primarily engaged through dialogic and exploratory interactions, especially via “Conversation across time”, reflecting the inquiry-oriented nature of the classroom activities. These patterns suggest that the observed learning benefits were supported not only by access to the AI tool, but by how its functionalities were enacted within specific instructional designs.
These findings underscore the importance of instructional mediation in AI-supported learning. Rather than operating as autonomous learning agents, AI applications appear most effective when embedded within coherent pedagogical frameworks that align the tool with curricular goals and teacher guidance.
4.3 Interpretation of motivation and engagement findings
In contrast to the academic performance results, no significant effects on motivation and engagement were observed. Although SB was actively used in both contexts during the pilot implementation, changes in motivational and engagement-related measures were modest and did not differ significantly between experimental and control groups. This finding diverges from a growing body of research reporting positive associations between AI-supported learning environments and students’ motivational and affective outcomes (Badarudin et al., 2024; Kuhail et al., 2023; Lee et al., 2022).
One possible explanation is that students’ interactions with SB were predominantly task- and content-focused, as indicated by the usage logs. While such interactions may support cognitive outcomes, they may be insufficient on their own to influence broader motivational dispositions, particularly over a short intervention period. Additionally, contextual factors such as the fixed three-weeks period of pilots, curricular demands, assessment pressures, and established classroom routines may have moderated the potential motivational impact of AI-supported activities. These findings suggest that motivational and engagement benefits may require longer-term exposure and more deliberate integration of autonomy-supportive and collaborative pedagogical strategies.
4.4 Interpretation of usage logs
The usage log analysis demonstrates that SB was adopted and used in both experimental contexts, but in different ways aligned with subject matter and implementation mode. In Physics, usage was primarily oriented toward concept clarification and revision during independent study, whereas in History, usage emphasized dialogic exploration, narrative engagement, and synthesis within structured classroom activities.
These findings highlight the value of incorporating usage analytics into evaluations of educational AI tools. By documenting not only whether students used the application but also how they engaged with its functionalities, usage data strengthen the interpretation of outcome measures and provide insight into the pedagogical mechanisms through which AI-supported tools may influence learning.
5. Conclusion
This study examined the educational impact of a purpose-built AI conversational study assistant in secondary school classrooms in Cyprus and Greece, with particular focus on students’ academic performance, motivation, and engagement. The findings indicate that SB use was consistently associated with significantly higher academic outcomes in both case studies. Students in the experimental groups demonstrated greater learning gains in Physics and achieved higher post-test scores in History compared to their peers in control groups.
Analysis of usage logs confirms that these outcomes occurred in contexts where students actively engaged with the application, albeit in different ways aligned with instructional design. In Physics, SB was primarily used for concept clarification and revision during independent study, while in History it supported dialogic, inquiry-oriented exploration under teacher guidance. These patterns suggest that the educational value of AI-supported study assistants lies not only in their availability, but in how they are integrated into subject-specific pedagogical practices.
With respect to motivation and engagement, no statistically significant differences were observed between experimental and control groups. Although SB was used across both contexts, short-term exposure and predominantly task-focused interactions may have limited its capacity to influence broader affective and behavioral outcomes. This finding underscores the importance of implementation duration, pedagogical design, and contextual alignment when aiming to support motivational dimensions of learning through AI tools.
Overall, the study suggests that purpose-built AI study assistants can function as effective cognitive supports for academic learning when embedded within regular instruction, while their impact on motivation and engagement may be more context-dependent and require sustained, intentionally designed pedagogical integration.
Several limitations should be considered when interpreting these results. First, the study employed a quasi-experimental design with intact classes, which limits causal inference. Although efforts were made to ensure group comparability, unmeasured pre-existing differences may have influenced outcomes. Second, sample sizes were relatively small, particularly in the motivation and engagement analyses, reducing statistical power and limiting generalizability. Third, the intervention duration was short; longer-term studies are needed to examine whether observed academic benefits persist over time and whether motivational and engagement-related effects emerge with sustained exposure. Fourth, although usage logs provided valuable insight into student access and interaction patterns, usage indicators were analyzed descriptively and were not linked to individual learning, motivation, or engagement outcomes. Future research should examine how variation in intensity and type of AI relates to learning processes and outcomes, while maintaining appropriate ethical safeguards. Finally, the study focused on two countries with relatively similar educational systems, which may constrain generalizability to other contexts. Future research should extend comparative analyses to more diverse educational settings and explore how different instructional models shape both AI usage patterns and educational impact.
Ethics and consent
Ethical approval for this study was obtained prior to the implementation of research in both participating countries. In Greece, the study was approved by the Ethics Committee of the Athena Research Center (reference number: 19–25/05/2023). In Cyprus, approval was granted by the Center for Educational Research and Evaluation (CERE), Ministry of Education, Sport and Youth (reference number: 07.30.002.002/140517).
Participation in the study was voluntary and written informed consent was obtained from participants and from the parents or legal guardians of minor participants prior to data collection.
