Please use this identifier to cite or link to this item:
https://repository.isls.org/handle/1/12744| Title: | Small vs. large language models: Comparing language models in generating productive failure math problems |
| Other Titles: | Small vs. Large Language Models: Comparing Language Models in Generating Productive Failure Math Problems |
| Authors: | Rahimi, Seyedahmad Esmaeiligoujar, Salah Ercan, Deniz Babaee, Maryam Gao, Ran |
| Keywords: | ICLS |
| Issue Date: | 2026 |
| metadata.dc.format.pagerange: | 1223-1227 |
| Publisher: | International Society of the Learning Sciences |
| Citation: | Rahimi, S., Esmaeiligoujar, S., Ercan, D., Babaee, M., & Gao, R. (2026). Small vs. large language models: Comparing language models in generating productive failure math problems. In Litts, B. K., DeLiema, D., Lee, C., Krist, S. Mawasi, A., Martin, L., & Kumpulainen, K. (Eds.), Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026 (pp. 1223-1227). International Society of the Learning Sciences. |
| Series/Report no.: | Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026 |
| Abstract: | Large language models (LLMs) such as GPT-4o can generate high-quality math problems but are resource-intensive and difficult to fine-tune. Small language models (SLMs) like Phi-3 may provide a more practical alternative, as they can run locally and be adapted for specific tasks. In this study, we generated 60 Productive Failure (PF) math problems, with half produced by GPT-4o and half by Phi-3. To ensure comparability, we controlled for grade level, Common Core standard, theme, interest, and prior knowledge. Trained raters evaluated the problems across five PF dimensions. Results showed that GPT-4o (M = 13.38; SD = 1.88) outperformed Phi-3 (M = 10.67; SD = 2.73), with a large effect size (d = 1.16). These findings indicate that although GPT-4o currently generates more effective PF problems, Phi-3 shows promising potential. With further fine-tuning and task-specific adaptation, SLMs like Phi-3 may offer a sustainable, cost-effective solution for educational problem generation. |
| Description: | SHORT |
| URI: | https://doi.org/10.22318/icls2026.113185 https://repository.isls.org/handle/1/12744 |
| ISBN: | 979-8-9906980-7-9 |
| ISSN: | 1819-0138 |
| Appears in Collections: | ISLS Annual Meeting 2026 |
Files in This Item:
| File | Size | Format | |
|---|---|---|---|
| ICLS2026_1223-1227.pdf | 293.72 kB | Adobe PDF | View/Open |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.