Please use this identifier to cite or link to this item: https://repository.isls.org/handle/1/12974
Title: Item calibration in reading assessment with LLMs: Human alignment and the masking effect
Other Titles: Item Calibration in Reading Assessment with LLMs: Human Alignment and the Masking Effect
Authors: Kong, Lingchen
Keywords: ICLS
Issue Date: 2026
metadata.dc.format.pagerange: 2543-2545
Publisher: International Society of the Learning Sciences
Citation: Kong, L. (2026). Item calibration in reading assessment with LLMs: Human alignment and the masking effect. In Litts, B. K., DeLiema, D., Lee, C., Krist, S. Mawasi, A., Martin, L., & Kumpulainen, K. (Eds.), Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026 (pp. 2543-2545). International Society of the Learning Sciences.
Series/Report no.: Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026
Abstract: This study evaluates large language models (LLMs) for calibrating item difficulty in grade 8 reading comprehension using 30 NAEP items. LLM-calibrated difficulty estimates were compared with human benchmarks, and masking was used to manipulate input information. Results show no significant differences across human-derived difficulty levels, indicating misalignment with human performance. Masking systematically reduced accuracy, particularly for harder items as more input text was obscured. These findings raise concerns about the validity of LLM-based item calibration.
Description: POSTER
URI: https://doi.org/10.22318/icls2026.617270
https://repository.isls.org/handle/1/12974
ISBN: 979-8-9906980-7-9
ISSN: 1819-0138
Appears in Collections:ISLS Annual Meeting 2026

Files in This Item:
File SizeFormat 
ICLS2026_2543-2545.pdf250.65 kBAdobe PDFView/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.