Methods for Analyzing Temporally Entangled Multimodal Data

Abstract

While the challenges of collecting multimodal data are becoming surmountable with the help of the rapid development of sensor technologies, the challenges of analyzing multimodality remain substantial. It is imperative that researchers explore ways to successfully integrate theoretical and methodological frameworks for analyzing multimodal interactions in CSCL contexts. We identified two main approaches to analyzing multimodal data in CSCL settings—triangulating and interleaving—and highlighted the remaining challenges to unfolding the dynamic interplay between different modes with the consideration of temporality. To tackle these challenges, we presented an empirical example of multimodal learning analysis that practically employed the multimodal matrix and ENA for operationalizing and visualizing temporally entangled multimodal interactions. This paper (1) extends the theoretical underpinnings of temporality in studies of learning processes in CSCL settings, and (2) provides empirical evidence that indicates the potential of the interleaving approach to capture the core of complex meaning-making processes.