Improving User Engagement and Learning Outcomes in LLM-Based Python Tutor: A Study of PACE

Abstract

Large Language Models (LLMs) are increasingly being adopted for educational applications, but sometimes, limited internet access and budget constraints restrict their accessibility. Small Language Models (SLMs) have emerged as viable alternatives, capable of providing effective tutoring in resource-constrained contexts. This paper introduces PACE (Python AI Companion for Enhanced Engagement), a system leveraging SLMs to deliver step-by-step guidance and adaptive feedback for teaching Python. An evaluation with varying levels of learners showed PACE’s effectiveness, achieving a System Usability Scale (SUS) score of 77.28. While participants were generally satisfied with its clarity and personalized feedback, they identified some areas for improvement, such as loss of context during lengthy conversations. This study examines (1) the PACE system’s effectiveness in programming education according to learners, (2) learners’ trust in PACE versus traditional resources, and (3) design recommendations to enhance engagement and learning outcomes. PACE contributes to advancing cost-effective, scalable programming education.

CCDS Authors

Ashraful Islam, PhD

Center DirectorDirector, Human-Computer Interaction Wing

References

1. 1.Virginia Braun, Victoria Clarke. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101[10.1191/1478088706qp063oa]
2. 2.David Wood, Jerome S. Bruner, Gail Ross. (1976). THE ROLE OF TUTORING IN PROBLEM SOLVING ^*. Journal of Child Psychology and Psychiatry, 17(2), 89–100[10.1111/j.1469-7610.1976.tb00381.x]
3. 3.Valerie J. Shute. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153–189[10.3102/0034654307313795]
4. 4.James R. Lewis. (2018). The System Usability Scale: Past, Present, and Future. International Journal of Human-Computer Interaction, 34(7), 577–590[10.1080/10447318.2018.1455307]
5. 5.Paul R. Pintrich. (2002). The Role of Metacognitive Knowledge in Learning, Teaching, and Assessing. Theory Into Practice, 41(4), 219–225[10.1207/s15430421tip4104_3]
6. 6.Dedre Gentner. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155–170[10.1016/s0364-0213(83)80009-3]
7. 7.Ilie Gligorea, Marius Cioca, Romana Oancea, Andra-Teodora Gorski, Hortensia Gorski, Paul Tudorache. (2023). Adaptive Learning Using Artificial Intelligence in e-Learning: A Literature Review. Education Sciences, 13(12), 1216[10.3390/educsci13121216]
8. 8.Emma Strubell, Ananya Ganesh, Andrew McCallum. (2020). Energy and Policy Considerations for Modern Deep Learning Research. Proceedings of the AAAI Conference on Artificial Intelligence, 34(09), 13693–13696[10.1609/aaai.v34i09.7123]
9. 9.Elham Mousavinasab, Nahid Zarifsanaiey, Sharareh Rostam Niakan Kalhori, Mahnaz Rakhshan, Leila Keikha, Marjan Ghazi Saeedi. (2018). Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1), 142–163[10.1080/10494820.2018.1558257]
10. 10.Rodrigo Smiderle, Sandro José Rigo, Leonardo Brandão Marques, Jorge Arthur Peçanha de Miranda Coelho, Patrícia A. Jaques. (2020). The impact of gamification on students’ learning, engagement and behavior based on their personality traits. Smart Learning Environments, 7(1)[10.1186/s40561-019-0098-x]
11. 11.Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adámek, Eric Malmi, Aliaksei Severyn. (2023). Teaching Small Language Models to Reason. , 1773–1781[10.18653/v1/2023.acl-short.151]
12. 12.Minju Park, So-Jung Kim, Seunghyun Lee, Soonwoo Kwon, Kyuseok Kim. (2024). Empowering Personalized Learning through a Conversation-based Tutoring System with Student Modeling. , 1–10[10.1145/3613905.3651122]
13. 13.Albert C. M. Yang, J.-J. Lin, Cheng-Yan Lin, Hiroaki Ogata. (2024). Enhancing python learning with PyTutor: Efficacy of a ChatGPT-Based intelligent tutoring system in programming education. Computers and Education Artificial Intelligence, 7, 100309[10.1016/j.caeai.2024.100309]
14. 14.Wenyue Hua, Lei Li, Shuyuan Xu, Li Chen, Yongfeng Zhang. (2023). Tutorial on Large Language Models for Recommendation. , 1281–1283[10.1145/3604915.3609494]
15. 15.Sankalan Pal Chowdhury, Vilém Zouhar, Mrinmaya Sachan. (2024). AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails. , 5–15[10.1145/3657604.3662041]
16. 16.Zhenwen Liang, Wenhao Yu, Tanmay Rajpurohit, Peter Clark, Xiangliang Zhang, Ashwin Kalyan. (2023). Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation. , 14384–14396[10.18653/v1/2023.emnlp-main.889]
17. 17.Shashank Sonkar, Naiming Liu, Debshila Basu Mallick, Richard G. Baraniuk. (2023). CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles. , 1941–1961[10.18653/v1/2023.findings-emnlp.130]
18. 18.Suqing Liu, Zezhu Yu, Feiran Huang, Yousef Bulbulia, Andreas Bergen, Michael Liut. (2024). Can Small Language Models With Retrieval-Augmented Generation Replace Large Language Models When Learning Computer Science?. , 388–393[10.1145/3649217.3653554]
19. 19.Qin Zhang, Ziqi Liu, Shirui Pan. (2025). The Rise of Small Language Models. IEEE Intelligent Systems, 40(1), 30–37[10.1109/mis.2024.3517792]
20. 20.Yulin Chen, Ning Ding, Hai-Tao Zheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou. (2024). Empowering Private Tutoring by Chaining Large Language Models. , 354–364[10.1145/3627673.3679665]
21. 21.Joel Manuel Prieto Andreu, António L. Palmeira. (2024). Quick review of pedagogical experiences using GPT-3 in education. Journal of Technology and Science Education, 14(2), 633[10.3926/jotse.2111]
22. 22.Mark Guzdial, Cindy E. Hmelo, Roland Hübscher, Kris Nagel, Wendy Newstetter, Sadhana Puntambekar, Amnon Shabo, Jennifer Turns, Janet L. Kolodner. (1997). Integrating and guiding collaboration. , 95–105[10.3115/1599773.1599784]
23. 23.Elena Verdú, Luisa M. Regueras, M. J. Verdu, J. P. de Castro, Dan Kohen-Vacs, Eran Gal, Miky Ronen. (2014). Intelligent tutoring interface for technology enhanced learning in a course of computer network design. , 1–7[10.1109/fie.2014.7044139]
24. 24.Barış Çukurbaşı, Mübin Kıyıcı. (2021). Instructional design and instructional effectiveness in virtual classrooms: Research trends and challenges. Australasian Journal of Educational Technology, 156–174[10.14742/ajet.6882]
25. 25.Cristina Niculescu. (2016). INTELLIGENT TUTORING SYSTEMS - TRENDS ON DESIGN, DEVELOPMENT AND DEPLOYMENT. eLearning and Software for Education[10.12753/2066-026x-16-218]
26. 26.Elena L. Grigorenko, Robert J. Sternberg. (1998). Dynamic testing.. Psychological Bulletin, 124(1), 75–111[10.1037//0033-2909.124.1.75]
27. 27.Marija Đokić Petrović, David Pritchard, Miloš Ivanović, Vladimir Cvjetković. (2016). IMI Python: Upgraded CS Circles web‐based Python course. Computer Applications in Engineering Education, 24(3), 464–480[10.1002/cae.21724]
28. 28.Geela Venise Firmalo Fabic, Antonija Mitrović, Kourosh Neshatian. (2018). Investigating the effects of learning activities in a mobile Python tutor for targeting multiple coding skills. Research and Practice in Technology Enhanced Learning, 13(1), 23[10.1186/s41039-018-0092-x]
29. 29.Harsh Kumar, David Rothschild, Daniel G. Goldstein, Jake M. Hofman. (2023). Math Education with Large Language Models: Peril or Promise?. SSRN Electronic Journal[10.2139/ssrn.4641653]
30. 30.Xi Wang, Laurence Aitchison, Maja Rudolph. (2023). LoRA ensembles for large language model fine-tuning. arXiv (Cornell University)[10.48550/arxiv.2310.00035]