Conference Paper2025Cited by 7

Improving User Engagement and Learning Outcomes in LLM-Based Python Tutor: A Study of PACE

Muhtasim Ibteda Shochcho, Mohammad Ashfaq Ur Rahman, Shadman Rohan, Ashraful Islam, Hasnain Heickal, AKM Mahbubur Rahman, M. Ashraful Amin, Amin Ahsan Ali

Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Association for Computing Machinery, pp. 1-12, ISBN: 9798400713958

  • Quality Education — Relevance: 95%
DOI: 10.1145/3706599.3720240

Abstract

Large Language Models (LLMs) are increasingly being adopted for educational applications, but sometimes, limited internet access and budget constraints restrict their accessibility. Small Language Models (SLMs) have emerged as viable alternatives, capable of providing effective tutoring in resource-constrained contexts. This paper introduces PACE (Python AI Companion for Enhanced Engagement), a system leveraging SLMs to deliver step-by-step guidance and adaptive feedback for teaching Python. An evaluation with varying levels of learners showed PACE’s effectiveness, achieving a System Usability Scale (SUS) score of 77.28. While participants were generally satisfied with its clarity and personalized feedback, they identified some areas for improvement, such as loss of context during lengthy conversations. This study examines (1) the PACE system’s effectiveness in programming education according to learners, (2) learners’ trust in PACE versus traditional resources, and (3) design recommendations to enhance engagement and learning outcomes. PACE contributes to advancing cost-effective, scalable programming education.

Keywords

  • learning outcomes
  • user engagement
  • large language models
  • Pace
  • Python programming education
  • programming education
  • automated feedback

CCDS Authors

Muhtasim Ibteda Shochcho
Muhtasim Ibteda Shochcho

Muhtasim Ibteda Shochcho

  • Intern, CCDS

Published at

CHI, BEA

Research Interests

Machine Learning, Natural Language Processing

Mohammad Ashfaq Ur Rahman
Mohammad Ashfaq Ur Rahman

Mohammad Ashfaq Ur Rahman

  • Intern, CCDS

Published at

CHI, BEA

Research Interests

Machine Learning, Natural Language Processing

Shadman Rohan
Shadman Rohan

Shadman Rohan

  • Research Assistant, CCDS

Published at

CHI, BEA, BLP

Research Interests

Machine Learning, Natural Language Processing

Ashraful Islam, PhD
Ashraful Islam

Ashraful Islam, PhD

  • Center Director, CCDS
  • Director, HCI Wing

Assistant Professor

Department of Computer Science and Engineering

Independent University, Bangladesh (IUB)

Published at

CHI, UIST, UbiComp, IJCNN, ICIP, HCII, ICDMW, BLP, IEEE Internet of Things Journal, JMIR, JMIR Human Factors, JMIR Formative Research, IEEE Access, Smart Agricultural Technology, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Human-Computer Interaction, AI for Social Good, AI for Public Health, AI for Impact

Hasnain Heickal
Hasnain Heickal

Hasnain Heickal

  • Collaborator, AI/ML Wing

Ph.D Candidate

UMass Amherst

Published at

CHI

Research Interests

Artificial Intelligence

AKM Mahbubur Rahman, PhD
AKM Mahbubur Rhman, PhD

AKM Mahbubur Rahman, PhD

  • Director, DS & IPT Wings

Associate Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

CHI, ECCV, WACV, PAKDD, LREC, IJCNN, ICPR, ICIP, HCII, BEA, Scientific Reports, Information Sciences

Prof. M Ashraful Amin, PhD
M Ashraful Amin, PhD

Prof. M Ashraful Amin, PhD

  • Founder Director, CCDS
  • Director, AI/ML, HCI & DS Wings

Professor

+1 more affiliation

Department of Information Sciences and Technology

George Mason University, USA

Published at

ACL, CHI, UbiComp, ECAI, WACV, PAKDD, IJCNN, ICPR, ICIP, HCII, CLEF, BLP, Scientific Reports, JMIR, IEEE Access, Smart Agricultural Technology, Frontiers in Computational Neuroscience, Diabetes Research and Clinical Practice, PLOS ONE

Research Interests

Machine Learning, Cognitive & Vision Science, Cybernetics, Surveillance & Security, ICT in Education, Health, & Agriculture, Human-Computer Interaction, Internet of Things, Robotics

Amin Ahsan Ali, PhD
Amin Ahsan Ali, PhD

Amin Ahsan Ali, PhD

  • Director, AI/ML Wing

Professor

Department of Computer Science and Engineering

Independent University, Bangladesh

Published at

ACL, CHI, UbiComp, ECCV, IJCAI, ECAI, WACV, IPSN, PAKDD, LREC, IJCNN, ICPR, ICIP, BEA, BLP, ICMLA, Scientific Reports, Scientific Data, Pattern Recognition, Information Sciences, Ad Hoc Networks, Drug and Alcohol Dependence, PLOS ONE

Research Interests

Artificial Intelligence, Machine Learning

References

  1. 1.Virginia Braun, Victoria Clarke. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101[10.1191/1478088706qp063oa]
  2. 2.David Wood, Jerome S. Bruner, Gail Ross. (1976). THE ROLE OF TUTORING IN PROBLEM SOLVING <sup>*</sup>. Journal of Child Psychology and Psychiatry, 17(2), 89–100[10.1111/j.1469-7610.1976.tb00381.x]
  3. 3.Valerie J. Shute. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153–189[10.3102/0034654307313795]
  4. 4.James R. Lewis. (2018). The System Usability Scale: Past, Present, and Future. International Journal of Human-Computer Interaction, 34(7), 577–590[10.1080/10447318.2018.1455307]
  5. 5.Paul R. Pintrich. (2002). The Role of Metacognitive Knowledge in Learning, Teaching, and Assessing. Theory Into Practice, 41(4), 219–225[10.1207/s15430421tip4104_3]
  6. 6.Dedre Gentner. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155–170[10.1016/s0364-0213(83)80009-3]
  7. 7.Ilie Gligorea, Marius Cioca, Romana Oancea, Andra-Teodora Gorski, Hortensia Gorski, Paul Tudorache. (2023). Adaptive Learning Using Artificial Intelligence in e-Learning: A Literature Review. Education Sciences, 13(12), 1216[10.3390/educsci13121216]
  8. 8.Emma Strubell, Ananya Ganesh, Andrew McCallum. (2020). Energy and Policy Considerations for Modern Deep Learning Research. Proceedings of the AAAI Conference on Artificial Intelligence, 34(09), 13693–13696[10.1609/aaai.v34i09.7123]
  9. 9.Elham Mousavinasab, Nahid Zarifsanaiey, Sharareh Rostam Niakan Kalhori, Mahnaz Rakhshan, Leila Keikha, Marjan Ghazi Saeedi. (2018). Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1), 142–163[10.1080/10494820.2018.1558257]
  10. 10.Rodrigo Smiderle, Sandro José Rigo, Leonardo Brandão Marques, Jorge Arthur Peçanha de Miranda Coelho, Patrícia A. Jaques. (2020). The impact of gamification on students’ learning, engagement and behavior based on their personality traits. Smart Learning Environments, 7(1)[10.1186/s40561-019-0098-x]
  11. 11.Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adámek, Eric Malmi, Aliaksei Severyn. (2023). Teaching Small Language Models to Reason. , 1773–1781[10.18653/v1/2023.acl-short.151]
  12. 12.Minju Park, So-Jung Kim, Seunghyun Lee, Soonwoo Kwon, Kyuseok Kim. (2024). Empowering Personalized Learning through a Conversation-based Tutoring System with Student Modeling. , 1–10[10.1145/3613905.3651122]
  13. 13.Albert C. M. Yang, J.-J. Lin, Cheng-Yan Lin, Hiroaki Ogata. (2024). Enhancing python learning with PyTutor: Efficacy of a ChatGPT-Based intelligent tutoring system in programming education. Computers and Education Artificial Intelligence, 7, 100309[10.1016/j.caeai.2024.100309]
  14. 14.Wenyue Hua, Lei Li, Shuyuan Xu, Li Chen, Yongfeng Zhang. (2023). Tutorial on Large Language Models for Recommendation. , 1281–1283[10.1145/3604915.3609494]
  15. 15.Sankalan Pal Chowdhury, Vilém Zouhar, Mrinmaya Sachan. (2024). AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails. , 5–15[10.1145/3657604.3662041]
  16. 16.Zhenwen Liang, Wenhao Yu, Tanmay Rajpurohit, Peter Clark, Xiangliang Zhang, Ashwin Kalyan. (2023). Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation. , 14384–14396[10.18653/v1/2023.emnlp-main.889]
  17. 17.Shashank Sonkar, Naiming Liu, Debshila Basu Mallick, Richard G. Baraniuk. (2023). CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles. , 1941–1961[10.18653/v1/2023.findings-emnlp.130]
  18. 18.Suqing Liu, Zezhu Yu, Feiran Huang, Yousef Bulbulia, Andreas Bergen, Michael Liut. (2024). Can Small Language Models With Retrieval-Augmented Generation Replace Large Language Models When Learning Computer Science?. , 388–393[10.1145/3649217.3653554]
  19. 19.Qin Zhang, Ziqi Liu, Shirui Pan. (2025). The Rise of Small Language Models. IEEE Intelligent Systems, 40(1), 30–37[10.1109/mis.2024.3517792]
  20. 20.Yulin Chen, Ning Ding, Hai-Tao Zheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou. (2024). Empowering Private Tutoring by Chaining Large Language Models. , 354–364[10.1145/3627673.3679665]
  21. 21.Joel Manuel Prieto Andreu, António L. Palmeira. (2024). Quick review of pedagogical experiences using GPT-3 in education. Journal of Technology and Science Education, 14(2), 633[10.3926/jotse.2111]
  22. 22.Mark Guzdial, Cindy E. Hmelo, Roland Hübscher, Kris Nagel, Wendy Newstetter, Sadhana Puntambekar, Amnon Shabo, Jennifer Turns, Janet L. Kolodner. (1997). Integrating and guiding collaboration. , 95–105[10.3115/1599773.1599784]
  23. 23.Elena Verdú, Luisa M. Regueras, M. J. Verdu, J. P. de Castro, Dan Kohen-Vacs, Eran Gal, Miky Ronen. (2014). Intelligent tutoring interface for technology enhanced learning in a course of computer network design. , 1–7[10.1109/fie.2014.7044139]
  24. 24.Barış Çukurbaşı, Mübin Kıyıcı. (2021). Instructional design and instructional effectiveness in virtual classrooms: Research trends and challenges. Australasian Journal of Educational Technology, 156–174[10.14742/ajet.6882]
  25. 25.Cristina Niculescu. (2016). INTELLIGENT TUTORING SYSTEMS - TRENDS ON DESIGN, DEVELOPMENT AND DEPLOYMENT. eLearning and Software for Education[10.12753/2066-026x-16-218]
  26. 26.Elena L. Grigorenko, Robert J. Sternberg. (1998). Dynamic testing.. Psychological Bulletin, 124(1), 75–111[10.1037//0033-2909.124.1.75]
  27. 27.Marija Đokić Petrović, David Pritchard, Miloš Ivanović, Vladimir Cvjetković. (2016). IMI Python: Upgraded CS Circles web‐based Python course. Computer Applications in Engineering Education, 24(3), 464–480[10.1002/cae.21724]
  28. 28.Geela Venise Firmalo Fabic, Antonija Mitrović, Kourosh Neshatian. (2018). Investigating the effects of learning activities in a mobile Python tutor for targeting multiple coding skills. Research and Practice in Technology Enhanced Learning, 13(1), 23[10.1186/s41039-018-0092-x]
  29. 29.Harsh Kumar, David Rothschild, Daniel G. Goldstein, Jake M. Hofman. (2023). Math Education with Large Language Models: Peril or Promise?. SSRN Electronic Journal[10.2139/ssrn.4641653]
  30. 30.Xi Wang, Laurence Aitchison, Maja Rudolph. (2023). LoRA ensembles for large language model fine-tuning. arXiv (Cornell University)[10.48550/arxiv.2310.00035]

Cited By

  1. 1.Xiaokang Lei, Ching Christie Pang, Yuyang Jiang, Xin Tong, Pan Hui. (2026). Co-designing Digital Humans for Online Learning: A Framework for Human-AI Pedagogical Integration. Lecture notes in computer science, 536–559[10.1007/978-3-032-30784-2_34]
  2. 2.Haiyang Xu, Yunhui Song, Black Sun, Niklas Elmqvist. (2026). EchoMind: Augmenting Recorded Presentations with Context-Grounded Conversational Agents and Collective Viewer Signals. Nordic Conference on Human-Computer Interaction, 1–20[10.1145/3829807.3829883]
  3. 3.Kashif Imteyaz, Isabel Lopez, Nakul Rajpal, Hunjun Shin, Saiph Savage. (2026). Upskilling with Generative AI: Practices and Challenges for Freelance Knowledge Workers. , 1–18[10.1145/3808045.3808068]
  4. 4.Fan Yang, Wanwan Xia, Wenting Qin. (2026). Exploring Large Language Models for University-Level Mathematics: A Comparative Study. Mathematics, 14(11), 1886[10.3390/math14111886]
  5. 5.Yasunobu Sumikawa, Akikazu Takada, Akira Ichinose, Ari Murakami, Yuki Toyono, Ryohei Ikejiri, Meiko Sakasegawa, Kaoru Sekine, Yuhei Yamauchi. (2026). Real-Time Discussion Transition and Main Theme Analyses for Teacher Support. Web Intelligence, 24(1-2), 20–36[10.1177/24056456261438366]
  6. 6.Evelyn Eika, Frode Eika Sandnes. (2026). Vibe-coding Interactive Teaching Tools: Narrowing the AI-Literacy Gap for Non-Technology Educators. International Conference on Information and Education Technology (ICIET), 530–534[10.1109/iciet69664.2026.11561665]
  7. 7.Zackary P. T. Sin, Miffy H. T. Cheung, Ye Jia, XiangZhi Eric Wang, Matthew W. H. Lai, Chen Li, Xiao Huang, Peter H. F. Ng, George Baciu, Qing Li. (2026). PockeTA: Virtual Teaching Assistant Platform with Course Context. Lecture notes in computer science, 41–55[10.1007/978-981-92-0042-9_4]
← All publications