Predicting dropout in professional online learning: a pilot study on self-regulated learning strategies from surveys and trace data
Main Article Content
Abstract
This pilot study investigates the relationship between self-regulated learning (SRL) strategies and early dropout in a largely underexplored professional online learning context. Forty-two professionals (N = 42) enrolled in a four-week course delivered via a workplace Learning Experience Platform (LXP) supporting self-directed, adaptive learning pathways. SRL was measured using self-reports and behavioural trace data collected during the first week. A dual-method analytic approach was employed: top-down models (logistic regression and decision trees) tested predefined SRL predictors, while bottom-up methods (hierarchical clustering and process mining) uncovered emergent learner profiles without prior assumptions. Trace-based SRL indicators, particularly the number of unique actions on the platform, time spent during week one, participation in the forum, and completion of an action plan, were significantly stronger predictors of dropout (p < .05) than self-reports, where only the goal-setting subscale was significant. Despite the small sample size limiting generalisability, the study demonstrates that early behavioural engagement patterns can reliably predict learner persistence, often outperforming self-reported SRL measures. These findings advance our understanding of SRL in professional online learning environments and provide a replicable analytic framework for integrating trace data and surveys. By identifying actionable SRL behaviours linked to course completion, this study contributes toward the development of adaptive workplace learning systems capable of detecting and mitigating early disengagement in real time.
Article Details
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Azevedo, R. (2005). Computer environments as metacognitive tools for enhancing learning. Educational Psychologist, 40(4), 193-197. https://doi.org/10.1207/s15326985ep4004_1
Azevedo, R., Taub, M., & Mudrick, N. V. (2017). Understanding and reasoning about real-time cognitive, affective, and metacognitive processes to foster self-regulation with advanced learning technologies. In Handbook of self-regulation of learning and performance (pp. 254-270). Routledge. https://doi.org/10.4324/9781315697048-17
Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., ... & Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17(1), 13. https://doi.org/10.1186/s41239-020-00187-1
Boud, D. (1995). Enhancing learning through self-assessment. Routledge. https://doi.org/10.4324/9781315041520
Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. The Internet and Higher Education, 27, 1-13. https://doi.org/10.1016/j.iheduc.2015.04.007
Cicchinelli, A., Veas, E., Pardo, A., Pammer-Schindler, V., Fessl, A., Barreiros, C., & Lindstädt, S. (2018, March). Finding traces of self-regulated learning in activity streams. In Proceedings of the 8th international conference on learning analytics and knowledge (pp. 191-200). https://doi.org/10.1145/3170358.3170381
CrossKnowledge. (n.d.). About us. Retrieved May 20, 2023, from https://www.crossknowledge.com/about-us/
Crossley, S., Paquette, L., Dascalu, M., McNamara, D. S., & Baker, R. S. (2016, April). Combining click-stream data with NLP tools to better understand MOOC completion. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 6-14). https://doi.org/10.1145/2883851.2883931
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29. https://doi.org/10.1177/0956797613504966
Cuyvers, K., Van den Bossche, P., & Donche, V. (2020). Self-regulation of professional learning in the workplace: a state of the art and future perspectives. Vocations and Learning, 13(2), 281-312. https://doi.org/10.1007/s12186-019-09236-x
Dabbagh, N., & Kitsantas, A. (2004). Supporting self-regulation in student-centered web-based learning environments. In International Journal on E-learning (Vol. 3, No. 1, pp. 40-47). Association for the Advancement of Computing in Education (AACE).
Endedijk, M. D., & Cuyvers, K. (2022). Self-regulation of professional learning: towards a new era of research. In Research approaches on workplace learning: insights from a growing field (pp. 219-237). Springer International Publishing. https://doi.org/10.1007/978-3-030-89582-2_10
Frankola, K. (2001). Why online learners drop out. American Society for Training and Development. Retrieved August 25, 2023, from http://www.c3l.uni-oldenburg.de/cde/support/readings/frankola.htm
Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349-360. https://doi.org/10.1093/poq/nfp031
Giannakos, M. N., Lee-Cultura, S., & Sharma, K. (2021). Sensing-based analytics in education: The rise of multimodal data enabled learning systems. IT Professional, 23(6), 31-38. https://doi.org/10.1109/MITP.2021.3089659
Greene, J. A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive and metacognitive processes while using computer-based learning environments. Educational Psychologist, 45(4), 203-209. https://doi.org/10.1080/00461520.2010.515935
Hadwin, A. F., Järvelä, S., & Miller, M. (2011). Self-regulated, co-regulated, and socially shared regulation of learning. Handbook of self-regulation of learning and performance, 30, 65-84.
Halawi, L., Clarke, A., & George, K. (2022). Decision Trees and Ensemble. In Harnessing the Power of Analytics (pp. 61-81). Springer International Publishing. https://doi.org/10.1007/978-3-030-89712-3_5
Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001
Hilpert, J. C., Greene, J. A., & Bernacki, M. (2023). Leveraging complexity frameworks to refine theories of engagement: Advancing self‐regulated learning in the age of artificial intelligence. British Journal of Educational Technology, 54(5), 1204-1221. https://doi.org/10.1111/bjet.13340
Hosmer Jr, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression. John Wiley & Sons. https://doi.org/10.1002/9781118548387
IMARC Group. (2023). E-learning market: Global industry trends, share, size, growth, opportunity and forecast 2023-2028. Retrieved from https://www.imarcgroup.com/e-learning-market
Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data clustering: A review. ACM Computing Surveys, 31(3), 264-323. https://doi.org/10.1145/331499.331504
Jansen, R. S., van Leeuwen, A., Janssen, J., Conijn, R., & Kester, L. (2020). Supporting learners’ self-regulated learning in Massive Open Online Courses. Computers & Education, 146, 103771. https://doi.org/10.1016/j.compedu.2019.103771
Jansen, R., van Leeuwen, A., Janssen, J., & Kester, L. (2020). A mixed method approach to studying self-regulated learning in MOOCs: combining trace data with interviews. Frontline Learning Research, 8(2), 35-64. https://doi.org/10.14786/flr.v8i2.539
Järvelä, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. Educational Psychologist, 48(1), 25-39. https://doi.org/10.1080/00461520.2012.748006
Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. International Review of Research in Open and Distributed Learning, 16(3), 341-358. https://doi.org/10.19173/irrodl.v16i3.2112
Kizilcec, R. F., & Halawa, S. (2015). Attrition and achievement gaps in online learning. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (pp. 57-66). Association for Computing Machinery. https://doi.org/10.1145/2724660.2724680
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in massive open online courses. Computers & Education, 104, 18-33. https://doi.org/10.1016/j.compedu.2016.10.001
Lee, D., Watson, S. L., & Watson, W. R. (2019). Systematic literature review on self-regulated learning in massive open online courses. Australasian Journal of Educational Technology, 35(1). https://doi.org/10.14742/ajet.3749
Li, Q., Baker, R., & Warschauer, M. (2020). Using clickstream data to measure, understand, and support self-regulated learning in online courses. The Internet and Higher Education, 45, 100727. https://doi.org/10.1016/j.iheduc.2020.100727
Littlejohn, A., & Margaryan, A. (2014). Technology-enhanced professional learning. International handbook of research in professional and practice-based learning, 1187-1212. https://doi.org/10.1007/978-94-017-8902-8_43
Littlejohn, A., & Milligan, C. (2015). Designing MOOCs for professional learners: Tools and patterns to encourage self-regulated learning. eLearning Papers, 42, 38-47. https://doi.org/10.6084/M9.FIGSHARE.866773
Littlejohn, A., Hood, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs: Motivations and self-regulated learning in MOOCs. The Internet and Higher Education, 29, 40-48. https://doi.org/10.1016/j.iheduc.2015.12.003
Maldonado-Mahauad, J. J., Pérez-Sanagustín, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from Big Data: Identifying self-regulated learning strategies in Massive Open Online Courses. Computers in Human Behavior, 90, 179-196. https://doi.org/10.1016/j.chb.2017.11.011
Malmberg, J., Järvenoja, H., & Järvelä, S. (2013). Patterns in elementary school students′ strategic actions in varying learning situations. Instructional Science, 41(5), 933-954. https://doi.org/10.1007/s11251-012-9262-1
Margaryan, A., Littlejohn, A., & Milligan, C. (2013). Self‐regulated learning in the workplace: strategies and factors in the attainment of learning goals. International Journal of Training and Development, 17(4), 245-259. https://doi.org/10.1111/ijtd.12013
Men, Q., Gimbert, B., & Cristol, D. (2023). The effect of self-regulated learning in online professional training. International Journal of Mobile and Blended Learning (IJMBL), 15(2), 1-17. https://doi.org/10.4018/IJMBL.318225
Milligan, C., & Littlejohn, A. (2014). Supporting professional learning in a massive open online course. International Review of Research in Open and Distributed Learning, 15(5), 197-213. https://doi.org/10.19173/irrodl.v15i5.1855
Moreno-Marcos, P. M., Muñoz-Merino, P. J., Maldonado-Mahauad, J., Pérez-Sanagustín, M., Alario-Hoyos, C., & Kloos, C. D. (2020). Temporal analysis for dropout prediction using self-regulated learning strategies in self-paced MOOCs. Computers & Education, 145, 103728. https://doi.org/10.1016/j.compedu.2019.103728
Muljana, P. S., & Luo, T. (2023). Pursuing Professional Learning by Using Social Media: How Do Instructional Designers Apply Self-Regulated Learning?. Research in Learning Technology, 31. https://doi.org/10.25304/rlt.v31.2934
Panadero, E., Kirschner, P. A., Järvelä, S., Malmberg, J., & Järvenoja, H. (2015). How individual self-regulation affects group regulation and performance: A shared regulation intervention. Small Group Research, 46(4), 431-454. https://doi.org/10.1177/1046496415591219
Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723-735. https://doi.org/10.1080/00313831.2015.1066436
Pintrich, P. R. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). https://doi.org/10.1037/t09161-000
Pintrich, P. R. (1999). The role of motivation in promoting and sustaining self-regulated learning. International Journal of Educational Research, 31(6), 459-470. https://doi.org/10.1016/S0883-0355(99)00015-4
R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
RStudio Team. (2023). RStudio: Integrated Development for R. Posit.
Reparaz, C., Aznárez-Sanado, M., & Mendoza, G. (2020). Self-regulation of learning and MOOC retention. Computers in Human Behavior, 111, 106423. https://doi.org/10.1016/j.chb.2020.106423
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J. C., & Müller, M. (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 1-8. https://doi.org/10.1186/1471-2105-12-77
Roll, I., & Winne, P. H. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7-12. https://doi.org/10.18608/jla.2015.21.2
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students' academic performance: a systematic review and meta-analysis. Psychological Bulletin, 138(2), 353. https://doi.org/10.1037/a0026838
Schunk, D. H. (2005). Self-regulated learning: The educational legacy of Paul R. Pintrich. Educational Psychologist, 40(2), 85-94. https://doi.org/10.1207/s15326985ep4002_3
Sharma, K., Papamitsiou, Z., & Giannakos, M. (2019). Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach. British Journal of Educational Technology, 50(6), 3004-3031. https://doi.org/10.1111/bjet.12854
Training Magazine. (2023). 2023 Training Industry Report. Training Magazine. Retrieved from https://trainingmag.com/trgmag-article/2023-training-industry-report
Tempelaar, D., Rienties, B., & Nguyen, Q. (2020). Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application. PLOS ONE, 15(6), e0233977. https://doi.org/10.1371/journal.pone.0233977
Viberg, O., Khalil, M., & Baars, M. (2020, March). Self-regulated learning and learning analytics in online learning environments: A review of empirical research. In Proceedings of the tenth international conference on learning analytics & knowledge (pp. 524-533). https://doi.org/10.1145/3375462.3375483
Valdiviezo, A. D., & Crawford, M. (2020). Fostering soft-skills development through learning experience platforms (LXPs). Handbook of Teaching with Technology in Management, Leadership, and Business, 312-321. https://doi.org/10.4337/9781789901658.00040
Van Halem, N., van Klaveren, C., Drachsler, H., Schmitz, M., & Cornelisz, I. (2020). Tracking Patterns in Self-Regulated Learning Using Students’ Self-Reports and Online Trace Data. Frontline Learning Research, 8(3), 140-163. https://doi.org/10.14786/flr.v8i3.497
Wan, Z., Compeau, D., & Haggerty, N. (2012). The effects of self-regulated learning processes on e-learning outcomes in organizational settings. Journal of Management Information Systems, 29(1), 307-340. https://doi.org/10.2753/MIS0742-1222290109
Wickham, H. (2016). Data analysis. In ggplot2: elegant graphics for data analysis (pp. 189-201). Springer International Publishing. https://doi.org/10.1007/978-3-319-24277-4_9
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. Metacognition in educational theory and practice, 277-304.
Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45(4), 267-276. https://doi.org/10.1080/00461520.2010.517150
Winne, P. H. (2011). A cognitive and metacognitive analysis of self-regulated learning: Faculty of education, Simon Fraser University, Burnaby, Canada. In Handbook of self-regulation of learning and performance (pp. 29-46). Routledge.
Winne, P. H. (2017). Learning analytics for self-regulated learning. Handbook of learning analytics, 241-249. https://doi.org/10.18608/hla17.021
Winne, P. H. (2020). Construct and consequential validity for learning analytics based on trace data. Computers in Human Behavior, 112, 106457. https://doi.org/10.1016/j.chb.2020.106457
Winne, P. H., & Hadwin, A. F. (2013). nStudy: Tracing and supporting self-regulated learning in the Internet. In International handbook of metacognition and learning technologies (pp. 293-308). Springer. https://doi.org/10.1007/978-1-4419-5546-3_20
Wong, J., Baars, M., de Koning, B. B., & Paas, F. (2021). Examining the use of prompts to facilitate self-regulated learning in Massive Open Online Courses. Computers in Human Behavior, 115, 106596. https://doi.org/10.1016/j.chb.2020.106596
Yeomans, M., & Reich, J. (2017, March). Planning prompts increase and forecast course completion in massive open online courses. In Proceedings of the seventh international learning analytics & knowledge conference (pp. 464-473). https://doi.org/10.1145/3027385.3027416
Ye, D., & Pennisi, S. (2022). Using trace data to enhance students' self-regulation: A learning analytics perspective. The Internet and Higher Education, 54, 100855. https://doi.org/10.1016/j.iheduc.2022.100855
Zimmerman, B. J., & Pons, M. M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23(4), 614-628. https://doi.org/10.3102/00028312023004614
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In Handbook of self-regulation (pp. 13-39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7
Zimmerman, B. J., & Moylan, A. R. (2009). Where metacognition and motivation intersect. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 299-316). Routledge.