Learning analytics is a young field of research (Baker & Siemens, 2014a; Baker & Yacef, 2009), that along with educational data mining has rapidly grown, driven by the availability of (large) sets of data on students’ learning and the interest in analysing these data for the purpose of improving students’ learning and learning experience. I do not make much of the difference between learning analytics and educational data mining here, but it is worth keeping in mind that there are differences between the two fields, even though they are closely related and draw on an very much overlapping research communities. Siemens and Baker (2012) identify the following differences:
- EDM researchers are more interested in automated methods for discovery, while LA is more interested in human-led, mixed-initiative methods for exploring educational data;•
- EDM is more construct-oriented, while LA researchers emphasize a more holistic view of learning and learners;
- Researchers in EDM develop methods for automatic adaptation of instruction, whereas LA researchers are developing applications that inform teachers, educators, and students. Hence the strong interest in LA on learning visualisations.
The focus of this paper is on the relation between learning analytics (and EDM) and learning research, in particular the kind of learning research practiced in the Learning Sciences (Sawyer, 2014). My intention is thus similar to the one of Baker and Siemens in their contribution the second edition of the Cambridge Handbook of the Learning Sciences (Baker & Siemens, 2014b): To contribute to a stronger tie between learning analytics and learning (sciences) research. However, different from Baker and Siemens I believe that important contributions from learning analytics to learning research are still a matter of the future. I argue that that while there is the potential for that, it is far from realized, even from being realized. In terms of Pasteur’s Quadrant (Stokes, 1997), I see learning analytics as currently falling into the category of pure applied research, whereas learning sciences can be see as use-inspired basic research, in which the focus is on advancing “the frontiers of understanding but also inspired by considerations of use” (Stokes, 1997, p. 74).
The main strategy I am following here is to develop some suggestions for how to make LA more relevant for foundational research on learning. I argue that the methods used in learning analytics (and EDM) have the potential to contribute to the applied as well as the foundational objectives of learning research. I further suggest that a more theory-oriented learning analytics can be more than an ‘addition’ to the ‘toolbox’ the learning researcher, that the ‘import’ could be more profound: It could change to a certain extent how we think about research methodology in the learning sciences.
The potential of Learning Analytics in learning research
The potential of learning analytics for the advancement of the learning research can in my opinion unfold along the four dimensions: (i) data quantity, (ii) longitudinal data, (iii) data from multiple levels, and (iv) data from many locations. In this section, I map these characteristics of data in learning analytics to modern conceptions of learning and main findings from learning research.
Quantity of Data
The size of data sets is the primary argument for the value of LA: ”One of the factors leading to the recent emergence of learning analytics is the increasing quantity of analysable educational data (…) Papers have recently been published with data from tens of thousands of students.” write Baker and Siemens (2014a, p. 254). Size is not only measured in number of students; the number of data points per student (captured in log files of learning applications and platforms, for instance) is another quantitative dimension. The Pittsburgh Science of Learning Center DataShop (Koedinger et al., 2010), for instance, stores detailed recordings of students’ interactions with carefully designed tutor software that records step-by-step problem solving operations.
There are a number of reasons why size is considered to matter. One is that the number of students is taken as useful for establishing the generalizability of findings—a statistical argument. Another is that the more data, the more ‘patterns’ can be found. The flip side to this is that the number of possible relations between variables increases exponentially with the number or variables included in the analysis (Council, 2013). More is needed than just data to ‘discover’ meaningful relations.
A third argument for the value of large data sets is that they allow us to identify ‘rare’ events: events/patterns that occur in only small numbers of students or only sporadically (e.g., Sabourin, Rowe, Mott, & Lester, 2011).This is particularly interesting if the rare events are defined apriori: events that theory predicts, but that are seldom occurring spontaneously, or are seldom observable because of interactions with other processes (or because of measurement issues). The inverse is interesting as well: Theory might not allow certain events to happen; if they happen, their appearance is interesting because this might not only be just a measurement error, or due to ‘chance’, but indicate a limitation of the theory; it might even render the theory downright wrong.
While all three aspects of data quantity are beneficial to learning research, the third aspect—rare event detection—deserves more attention. It is the one least often considered, but it can contribute to make learning sciences more theory-guided, and it can help to bridge the gap between qualitative and quantitative learning research. In qualitative research, the frequency with which an event occurs is not automatically identified with the importance of the event; in many cases, important events are rare. An example from learning research is conceptual change, which occurs rarely, but when it occurs has profound effects on students’ understanding (diSessa, 2006).
Longitudinal Learning Data
Learning needs time.Learning in schools and universities requires often multiple skills—such as mathematical and writing skills—to master complex, hierarchically structured subject matter. In science education, for instance, the hierarchical nature of the subject knowledge also leads to the subject being an intricate association of concepts where deep learning of some basic concepts require comprehension of other basic concepts (Fergusson-Hessler & de Jong, 1987). Theoretical accounts for the depth and extend it takes to comprehend scientific concepts have been suggested from a cognitive psychology perspective and from a socio-cultural perspective. From the cognitive psychology perspective, one line of argument is that learning science can be seen as developing a form of expertise, and that any form of real expertise in cognitively demanding areas requires years of learning (the magic number is 10 years, plus/minus 2), as evidenced by novice-expert research, see (K. A. Ericsson, Charness, Feltovich, & Hoffman, 2006) for a comprehensive overview. The currently best elaborated cognitive model of expertise development in the cognitive tradition is probably Ericsson’s Deliberate Practice theory (K. Anders Ericsson, Krampe, & Tesch-Römer, 1993). The reason why learning takes long in this model is the incremental nature of the underlying cognitive learning/change mechanisms (chunking, proceduralization).
Another cognitive account, and one more specific to science education than general models of expertise development, is Chi’s and Slotta’s Ontology Shift theory (e.g., Chi, Slotta & de Leeuw, 1994). On this account, learning scientific concepts is hard and everyday concepts are resistant to change because scientific understanding requires in many cases a change in an ontological category. A classical example is the concept of heat, where students often see heat as a property of matter, whereas in physics it is seen in process terms, as the average velocity of particles. In this theory, the reason that learning stretches often over longer times is that while the ontology change itself can be fairly rapid, it needs often extended time (under current conditions of science learning) before students become sufficiently aware of the limitations of the initial ontology and are ready to accept an alternative one.
Tracking learning that stretches over months and years—another example for this would be the development of second language skills—is very rarely done in learning research. One reason are the costs, and the logistics, of performing such research. But the costs are being substantially lowered as learning analytics methods find their place in schools and universities. It would be of tremendous benefit if such data could be made available to researchers, and their acquisition planned in coordination with research projects. Methods for process mining are particularly relevant in this context (Reimann, 2009). Not only would this help to conduct specific projects that study long-term learning, it would also change the way we think about the nature of projects in learning research: From short-term interventions with immediate effects assessment to longer-duration interventions with continuous, long-durations effects (and side-effects!) monitoring. A variant of this kind of research we see developing with improvement research (Bryk, 2015), and the continuous use of data for decision making (Mandinach, 2012).
Data from Learning on Multiple Levels - Learning is complex
Learning does not only place over long durations, but on other levels of analysis is happening within seconds and even milliseconds. Nathan and Alibali (2010) distinguish between learning in milliseconds and below (biological), seconds (cognitive), minutes to hours (rational), days to months (sociocultural), and years and beyond (organizational). This can be seen as an expression of strictly different kinds of learning, but more productively it may be seen as an expression of the fact that learning takes place at multiple levels at the same time. We can see learning ‘events’ as being produced by a complex, multi-layered system, with minimally three levels: A biological stratum with neurophysiological processes, a cognitive stratum (rational thinking, knowledge) , and a socio-cultural stratum (tools, practices). These strata, or levels, are set in relation to each other by processes of emergence (Sawyer, 2005).
The concept of emergence as used here is relational: It refers to the phenomenon that wholes (entities, agents, organisms, organisations) have properties that cannot be found in any of their parts. An emergent property “is one that is not possessed by any of the parts individually and that would not be possessed by the full set of parts in the absence of a structuring set of relations between them.” (Elder-Vass, 2010, p. 17). A key aspect of (relational) emergence is therefore the organization of the parts, how the parts are set in relation to each other, how the whole is structured. Not all properties of an object are emergent; some will be resultant properties. For instance, most objects have mass, which is an resultant property: the mass of the whole is the sum of parts’ masses. Some objects have colour, which is an emergent property; it is dependent on the organization of the objects’ parts.
If we conceive of learning as a complexity phenomenon (Kapur et al., 2007), then learning needs not only be studied at multiple levels, but the analysis of the relation between the levels—the nature of the emergence—must take center stage. This requires not only to ask what affects learning over time, but also how learning is constituted at each moment in time: Which configurations of neural, cognitive, motivational, emotional, social and contextual processes/elements give rise to a ‘learning event’? Answering the latter question requires appropriate instrumentation, and appropriate analytical methods. The methods cannot be (only) variants of the General Linear Model (e.g., regression models, including so-called ‘structural’ or ‘causal’ variants), amongst other reasons because these are not appropriate for non-linear complex systems, for systems that transform themselves or get transformed. Instead, methods for the analysis of non-linear systems will be needed (e.g., van Geert, 1998), and methods that can be used to describe relations between parts, in particular graph-theoretical methods such as Social Network Analysis (Burt, Kilduff, & Tasselli, 2013). Learning analytics and educational data mining can play a key role in advancing the learning sciences by bringing about such methodological advances and by making them usable for learning researchers. These includes, but should not be confined to, methods for recording bio-signals, learning behavior and the cognitive-motivational processes causing them, as well as the social dimension of learning in great detail, with high precision, repeatedly and frequently, if not continuously.
Data from Learning in Many Contexts - Learning is Distributed
The methods being developed in learning analytics and educational data mining to capture aspects of students’ behaviour—and physiological and emotional parameters that go along with behaviour—not only over time, but also across locations is tremendously valuable for research. This because learning is situated: It is highly dependent on the resources available to the learner in specific contexts. Not only is learning happening (quasi-)synchronously across multiple levels, it is also distributed over the socio-physical environment—the situation—the learner finds herself in (Sawyer & Greeno, 2009). As Greeno and others have argued, any analysis of learning will be incomplete if it does not (also) conceptualise learning as a socio-cultural practice, as an activity system that stretches far beyond the somato-physical boundaries of the cranium and the body.
Such an understanding of learning practices is necessary for theoretical as well as pedagogical purposes. For the purpose of theory development, an understanding of the socio-material practices around knowledge objects contributes to de-mystifying the process of learning—how is it possible to learn something genuinely new?— and of idea and knowledge creation more generally (Prawat, 1999). As the entanglement of cognitive work with physical, symbolic and social resources becomes ever better documented and understood—in general (e.g., Clark, 2011) and for specific areas such as scientific research (e.g., Latour & Woolgar, 1986)—it becomes clear that a theory of learning, creativity and idea generation will need to be grounded not only in psychology, but also in sociology, organization science, and semiotics. Any specific study will need to capture knowledge practices in a comprehensive sense.
The fact that with learning analytics methods behavioural, interactional, and increasingly even some physiological parameters of students’ ‘learning’ activities can be captured across locales and contexts constitutes an essential prerequisite for researching learning-in-context at scale. Learning analytics methods will need to become substantially more sophisticated to become really useful for studying learning-in-context, though. It is not sufficient to keep track of students’ activities (and related parameters) alone; in addition, the context needs to be described and logged as well. This is easier said than done; just think of the many artefacts and tools that students use on average on every day of a semester: at school/uni, at home, while commuting. Along with technical advancements for capturing aspects of students’ behaviour and experience, a main focus of research in learning analytics should therefore be to develop languages, and standards, for describing the context within which behaviour and experience arise, and for describing the relation between the learners and the social, physical and symbolic aspects of learning context.
In summary, I argue that there lies a huge potential in learning analytics to advance learning research, and that in order to realize this potential learning analytics researchers should devote more attention to (finding) rare learning events, to focus more on long-term learning, to make more of the fact that learning can be recorded on multiple levels of a complex system (the human learner), and to develop methods for capturing the context in which learning activities occur. None of this can be done without building on theory, on conceptualizations of learning and cognition. Theory is essential, and it is important to repeat what two of the key researchers write: ”The theory-oriented perspective marks a departure of EDM and LA from technical approaches that use data as their sole guiding point…” (Baker & Siemens, 2014b, p. 256/257). Suggestions such as made by Anderson (2008) that big data will render the scientific method obsolete not only express a deep misunderstanding of what the method is about, they are also committing the logical (and ethical) error of using descriptions of the past as prescriptions for the future.
Anderson, C. . (2008). The end of theory: The data deluge makes the scientific method obsolete. Wired Magazin. Retrieved 14 December, 2015, from http://www.wired.com/2008/06/pb-theory/
Baker, R., & Siemens, G. (2014a). Educational data mining and learning analytics. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences (2nd ed., pp. 253-274). New York: Cambridge University Press.
Baker, R., & Siemens, G. (2014b). Learning analytics and educational data mining. In R. K. Sawyer (Ed.), Cambridge Handbook of the Leaning Sciences (2nd ed., pp. 253-272). New York: Cambridge University Press.
Baker, R., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future vision. JEDM - Journal of Educational Data Mining, 1(1), 3-17.
Bryk, A. S. (2015). 2014 AERA Distinguished Lecture: Accelerating How We Learn to Improve. Educational Researcher.
Burt, Ronald S., Kilduff, Martin, & Tasselli, Stefano. (2013). Social network analysis: Foundations and frontiers on advantage. Annual Review of Psychology, 64, 527-547.
Clark, A. (2011). Supersizing the mind. Embodiment, action, and cognitive extension. Oxford, UK: Oxford University Press.
Council, National Research. (2013). Frontiers in Massive Data Analysis. Washington, D.C.: The National Academic Press.
diSessa, A.A. (2006). A history of conceptual change research: Threads and fault lines. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences.
Elder-Vass, Dave. (2010). The causal power of social structures. Cambridge, UK: Cambridge University Press.
Ericsson, K. A., Charness, N., Feltovich, P., & Hoffman, R.B. (Eds.). (2006). The Cambridge Handbook of Expertise and Expert Performance. New York: Cambride University Press.
Ericsson, K. Anders, Krampe, Ralf Th., & Tesch-Römer, Clemens. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406.
Fergusson-Hessler, M.G.M., & de Jong, T. (1987). On the quality of knowledge in the field of electricity and magnetism. American Journal of Physics, 55, 492-497.
Kapur, M., Hung, D., Jacobson, M.J., Voiklis, J., Kinzer, C. K., & Victor, Chen Der-Thang. (2007). Emergence of learning in computer-supported, large-scale collective dynamics: A research agenda Proceedings of the International Conference on Computer-supported Collaborative Learning (CSCL2007). New Brunswick, NJ.
Koedinger, K R, Baker, R S J D , Cunningham, K, Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. In C. Robero, S. Ventura, M. Pechenizkiy & R. Baker (Eds.), Handbook of educational data mining (pp. 43-56). Boca Raton, FL.: Chapman&Hall/CRC.
Latour, B., & Woolgar, S. (1986). Laboratory life: The construction of scientific facts (2nd ed.). Princeton: Princeton University Press.
Mandinach, E. B. (2012). A perfect time for data use: Using data-driven decision making to inform practice. Educational Psychologist, 47(2), 71-85.
Nathan, M.J., & Alibali, Martha Wagner. (2010). Learning Sciences. Wiley Interdisciplinary Reviews:Cognitive Science, 1(3), 329-345.
Prawat, R. S. (1999). Dewey, Peirce, and the Learning Paradox. American Educational Research Journal, 36, 47-76.
Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-supported Collaborative Learning, 4, 239-257.
Sabourin, J., Rowe, J., Mott, B., & Lester, J. (2011). When off-task is on-task: The affective role of off-task behavior in narrative-centered learning environments. . Paper presented at the Proceedings of the 15th International Conference on Artificial Intelligence in Educatoin.
Sawyer, R. K. (2005). Social emergence. Societies as complex systems. Cambridge, UK: Cambridge University Press.
Sawyer, R. K. (Ed.). (2014). The Cambridge Handbook of the Learning Sciences (2nd ed.). New York: Cambride University Press.
Sawyer, R. K., & Greeno, J.G. (2009). Situativity and learning. In P. Robbins & M. Aydede (Eds.), The cambridge handbook of situated cognition (pp. 347-367). New York, NY: Cambridge University Press.
Siemens, G., & Baker, R.S.J. d. (2012). Learning analytics and educational data mining: Towards communication and collaboration. Paper presented at the Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK 2012).
Stokes, D.E. (1997). Pasteur's quadrant: Basic science and technological innovation. Washington, DC: Brookings Institution Press.
van Geert, Paul. (1998). A dynamic systems model of basic developmental mechanisms: Piaget, Vygotsky, and beyond. Psychological Review, 105, 634-677.