Please use this identifier to cite or link to this item:
Title: Rethinking Rater Effects When Using Teacher Observation Protocols
Authors: Chen, Ying
Yim, Rachel A.
Kogen, Richard
Stieff, Mike
Superfine, Alison Castro
Keywords: Teaching and Teacher Learning
Issue Date: Jun-2020
Publisher: International Society of the Learning Sciences (ISLS)
Citation: Chen, Y., Yim, R. A., Kogen, R., Stieff, M., & Superfine, A. C. (2020). Rethinking Rater Effects When Using Teacher Observation Protocols. In Gresalfi, M. and Horn, I. S. (Eds.), The Interdisciplinarity of the Learning Sciences, 14th International Conference of the Learning Sciences (ICLS) 2020, Volume 4 (pp. 2022-2029). Nashville, Tennessee: International Society of the Learning Sciences.
Abstract: Analysis of teacher practice using observation protocols is often adversely impacted by construct-irrelevant sources of variance. Identifying and understanding these factors and controlling their effects are crucial for improving the validity and reliability of observation protocols. Here, we describe the application of many-faceted Rasch model (MFRM) to account for these effects in learning sciences research. We used data from 172 high school chemistry classroom videos to illustrate the utility of MFRM to model latent constructs related to raters. The analysis shows our raters can display high internal consistency with varying levels of bias despite intensive training. Such differences can significantly impact claims about teacher practice. We demonstrate how MFRM can incorporate these differences into a model that accounts for rater biases and other threats to validity. Additionally, the model allows for the incorporation of individual raters’ expertise into measures of teacher practice by accounting for identified variability without removing it.
Appears in Collections:ICLS 2020

Files in This Item:
File SizeFormat 
2022-2029.pdf305.97 kBAdobe PDFView/Open

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.