Please use this identifier to cite or link to this item:
https://repository.isls.org/handle/1/12974| Title: | Item calibration in reading assessment with LLMs: Human alignment and the masking effect |
| Other Titles: | Item Calibration in Reading Assessment with LLMs: Human Alignment and the Masking Effect |
| Authors: | Kong, Lingchen |
| Keywords: | ICLS |
| Issue Date: | 2026 |
| metadata.dc.format.pagerange: | 2543-2545 |
| Publisher: | International Society of the Learning Sciences |
| Citation: | Kong, L. (2026). Item calibration in reading assessment with LLMs: Human alignment and the masking effect. In Litts, B. K., DeLiema, D., Lee, C., Krist, S. Mawasi, A., Martin, L., & Kumpulainen, K. (Eds.), Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026 (pp. 2543-2545). International Society of the Learning Sciences. |
| Series/Report no.: | Proceedings of the 20th International Conference of the Learning Sciences - ICLS 2026 |
| Abstract: | This study evaluates large language models (LLMs) for calibrating item difficulty in grade 8 reading comprehension using 30 NAEP items. LLM-calibrated difficulty estimates were compared with human benchmarks, and masking was used to manipulate input information. Results show no significant differences across human-derived difficulty levels, indicating misalignment with human performance. Masking systematically reduced accuracy, particularly for harder items as more input text was obscured. These findings raise concerns about the validity of LLM-based item calibration. |
| Description: | POSTER |
| URI: | https://doi.org/10.22318/icls2026.617270 https://repository.isls.org/handle/1/12974 |
| ISBN: | 979-8-9906980-7-9 |
| ISSN: | 1819-0138 |
| Appears in Collections: | ISLS Annual Meeting 2026 |
Files in This Item:
| File | Size | Format | |
|---|---|---|---|
| ICLS2026_2543-2545.pdf | 250.65 kB | Adobe PDF | View/Open |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.