Please use this identifier to cite or link to this item: https://repository.isls.org/handle/1/12744
Title: Small vs. large language models: Comparing language models in generating productive failure math problems
Other Titles: Small vs. Large Language Models: Comparing Language Models in Generating Productive Failure Math Problems
Authors: Rahimi, Seyedahmad
Esmaeiligoujar, Salah
Ercan, Deniz
Babaee, Maryam
Gao, Ran
Keywords: ICLS
Issue Date: 2026
metadata.dc.format.pagerange: 1223-1227
Publisher: International Society of the Learning Sciences
Citation: Rahimi, S., Esmaeiligoujar, S., Ercan, D., Babaee, M., & Gao, R. (2026). Small vs. large language models: Comparing language models in generating productive failure math problems. In Litts, B. K., DeLiema, D., Lee, C., Krist, S. Mawasi, A., Martin, L., & Kumpulainen, K. (Eds.), Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026 (pp. 1223-1227). International Society of the Learning Sciences.
Series/Report no.: Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026
Abstract: Large language models (LLMs) such as GPT-4o can generate high-quality math problems but are resource-intensive and difficult to fine-tune. Small language models (SLMs) like Phi-3 may provide a more practical alternative, as they can run locally and be adapted for specific tasks. In this study, we generated 60 Productive Failure (PF) math problems, with half produced by GPT-4o and half by Phi-3. To ensure comparability, we controlled for grade level, Common Core standard, theme, interest, and prior knowledge. Trained raters evaluated the problems across five PF dimensions. Results showed that GPT-4o (M = 13.38; SD = 1.88) outperformed Phi-3 (M = 10.67; SD = 2.73), with a large effect size (d = 1.16). These findings indicate that although GPT-4o currently generates more effective PF problems, Phi-3 shows promising potential. With further fine-tuning and task-specific adaptation, SLMs like Phi-3 may offer a sustainable, cost-effective solution for educational problem generation.
Description: SHORT
URI: https://doi.org/10.22318/icls2026.113185
https://repository.isls.org/handle/1/12744
ISBN: 979-8-9906980-7-9
ISSN: 1819-0138
Appears in Collections:ISLS Annual Meeting 2026

Files in This Item:
File SizeFormat 
ICLS2026_1223-1227.pdf293.72 kBAdobe PDFView/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.