Rethinking Rater Effects When Using Teacher Observation Protocols

Abstract

Analysis of teacher practice using observation protocols is often adversely impacted by construct-irrelevant sources of variance. Identifying and understanding these factors and controlling their effects are crucial for improving the validity and reliability of observation protocols. Here, we describe the application of many-faceted Rasch model (MFRM) to account for these effects in learning sciences research. We used data from 172 high school chemistry classroom videos to illustrate the utility of MFRM to model latent constructs related to raters. The analysis shows our raters can display high internal consistency with varying levels of bias despite intensive training. Such differences can significantly impact claims about teacher practice. We demonstrate how MFRM can incorporate these differences into a model that accounts for rater biases and other threats to validity. Additionally, the model allows for the incorporation of individual raters’ expertise into measures of teacher practice by accounting for identified variability without removing it.