Please use this identifier to cite or link to this item: https://repository.isls.org/handle/1/11963
Title: Capturing Collaborative Competency with GPT-4o and ENA
Authors: Lee, Yoonjae
Kwon, Christine
Seoh, Sarah
Gweon, Gahgene
Stamper, John
Rosé, Carolyn P.
Keywords: CSCL
Issue Date: 2025
Publisher: International Society of the Learning Sciences
Citation: Lee, Y., Kwon, C., Seoh, S., Gweon, G., Stamper, J., & Rosé, C. P. (2025). Capturing Collaborative Competency With Gpt-4o and Ena. In Oshima, J., Chen, B., Vogel, F., & Järvelä, J. (Eds.), Proceedings of the 18th International Conference on Computer-Supported Collaborative Learning - CSCL 2025 (pp. 72-80). International Society of the Learning Sciences.
Series/Report no.: Proceedings of the 18th International Conference on Computer-Supported Collaborative Learning - CSCL 2025
Abstract: Collaboration is a critical learning skill employed across many educational domains. Since moments of collaborative competencies frequently occur in student discourse, there has been a growing body of work in automatic collaborative process analysis. With recent advancements in Large Language Models (LLMs), it became feasible to automate the collaborative process analysis through simple prompting methods. In this study, we develop CoComTag, an LLM-powered approach using GPT-4o that captures students' collaborative competency using four prompting techniques. Our findings demonstrate that CoComTag shows substantial agreement with humans (kappa of 0.67). Such a result shows that LLM yields comparable performance with prior studies using only prompting strategies without additional fine-tuning. Next, using qualitative error analysis and epistemic network analysis (ENA), we examine how CoComTag differs from humans in detecting collaborative competency. Our analysis shows that ENA provides visual cues on where collaborative competencies co-occur, revealing the differences between LLM and human annotation patterns.
Description: Long Paper
URI: https://doi.org/10.22318/cscl2025.611229
https://repository.isls.org//handle/1/11963
Appears in Collections:ISLS Annual Meeting 2025

Files in This Item:
File SizeFormat 
CSCL2025_72-80.pdf469.02 kBAdobe PDFView/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.