Insufficient Effort Responding in Surveys Assessing Self-Regulated Learning: Nuisance or Fatal Flaw?
Main Article Content
Abstract
Despite concerns about their validity, self-report surveys remain the primary data collection method in the research of self-regulated learning (SRL). To address some of these concerns, we took a data set comprised of college students’ self-reported beliefs and behaviours related to SRL, assessed across three surveys, and examined it for instance of a specific threat to validity, insufficient effort responding (IER; Huang, Curran, Keeny, Poposki, & DeShon, 2012). Using four validated indicators of IER, we found the rate of IER to vary between 12-16%. Critically, while we found that students characterised as inattentive and attentive differed in some basic descriptive statistics, the inclusion of inattentive students within the data set did not alter more substantial inferences or conclusions drawn from the data. This study provides the first direct examination of the impact of respondents’ attention on the validity of SRL data generated from self-report surveys.
Article Details
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Azevedo, R., Taub, M., & Mudrick, N.V. (2018). Using multi-channel trace data to infer and foster self-regulated learning between humans and advanced learning technologies. In D. Schunk & Greene, J.A (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 254-270). New York, NY: Routledge.
Bandura, A. (2006). Guide for constructing self-efficacy scales. In Self-efficacy beliefs of adolescents (pp. 307–337). https://doi.org/10.1017/CBO9781107415324.004
Beach, D. A. (1989). Identifying the random responder. Journal of Psychology: Interdisciplinary and Applied, 123(1), 101–103. https://doi.org/10.1080/00223980.1989.10542966
Biswas, G., Baker, R,, & Paquette, L. (2018). Data mining methods for assessing self-regulated learning. In D. H. Schunk & J. A. Greeene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 388 - 403). New York: Routledge.
Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218-229. https://doi.org/10.1037/pspp0000085
Butler, D. L. (2002). Qualitative approaches to self regulated learning: Contributions and challenges. Educational Psychologist, 37, 59–63. https://doi.org/10.1207/S15326985EP3701
Carver, R.P. (1992). Reading rate: Theory, practice, and practical mplications. Journal of Reading, 36(2), 84-95.
Choi, J. N., & Moran, S. V. (2010). Why not procrastinate? Development and validation of a new active procrastination scale why not procrastinate. The Journal of Social Psychology, 149, 37–41. https://doi.org/10.3200/SOCP.149.2.195-212
Cleary, T. J, & Callan, G. (2018). Assessing self-regulated learning using microanalytic methods. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.
Cleary, T. J., & Zimmerman, B. J. (2004). Self-regulation empowerment program: A school-based program to enhance self-regulated and self-motivated cycles of student learning. Psychology in the Schools, 41(5), 537–550. https://doi.org/10.1002/pits.10177
Costa, P. T., & McCrae, R. R. (2008). The revised NEO personality inventory (NEO-PI-R). In The SAGE Handbook of Personality Theory and Assessment: Volume 2 - Personality Measurement and Testing (pp. 179–198). London: SAGE Publications Ltd. https://doi.org/10.4135/9781849200479.n9
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006
DeSimone, J. A., & Harms, P. D. (2017). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 1–19. https://doi.org/10.1007/s10869-017-9514-9
Diedenhofen, B., & Musch, J. (2016). cocron : A web interface and R package for the statistical comparison of Cronbach ’ s alpha coefficients. International Journal of Internet Science, 11(1), 51–60. http://www.ijis.net/ijis11_1/ijis11_1_diedenhofen_and_musch.pdf
Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(3), 231–264. https://doi.org/10.1007/s11409-008-9029-x
Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the short grit scale ( Grit – S ). Journal of Personality Assessment, 3891, 166–174. https://doi.org/10.1080/00223890802634290
Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251. https://doi.org/10.3102/0013189X15584327
Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2016). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 1–17. https://doi.org/10.1007/s10869-016-9479-0
Dweck, C. (2012). Mindset: How You Can Fulfil Your Potential. London: Robinson.
Feldt, L. S., Woodruff, D. J., & Salih, F. a. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93–103. https://doi.org/10.1177/014662168701100107
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8
Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845. https://doi.org/10.1037/a0038510
Hulleman, C. S., Durik, A. M., Schweigert, S. A., & Harackiewicz, J. M. (2008). Task values, achievement goals, and interest: An integrative analysis. Journal of Educational Psychology, 100(2), 398–416. https://doi.org/10.1037/0022-0663.100.2.398
Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129. https://doi.org/10.1016/j.jrp.2004.09.009
Karabenick, S. A., & Zusho, A. (2011). Examining approaches to research on self-regulated learning: conceptual and methodological considerations. Metacognition and Learning, 10(1), 151–163. https://doi.org/10.1007/s11409-015-9137-3
Macan, T. H. (1994). Time management: Test of a process model. Journal of Applied Psychology, 79(3), 381–391. https://doi.org/10.1037/0021-9010.79.3.381
Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61–83. https://doi.org/10.1016/j.jrp.2013.09.008
McCardle, L., & Hadwin, A. F. (2015). Using multiple, contextualized data sources to measure learners’ perceptions of their self-regulated learning. Metacognition and Learning, 10(1), 43–75. https://doi.org/10.1007/s11409-014-9132-0
McKibben, W. B., & Silvia, P. J. (2017). Evaluating the distorting effects of inattentive responding and social desirability on self-report scales in creativity and the arts. Journal of Creative Behavior, 51(1), 57–69. https://doi.org/10.1002/jocb.86
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
Nichols, D. S., Greene, R. L., & Schmolck, P. (1989). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: Rationale, development, and empirical trials. Journal of Clinical Psychology, 45(2), 239–250. https://doi.org/10.1002/1097-4679(198903)45:2<239::AID-JCLP2270450210>3.0.CO;2-1
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8(APR), 1–28. https://doi.org/10.3389/fpsyg.2017.00422
Pajares, F. (2007). Motivational role of self-efficacy beliefs in self-regulated learning. In B. J. Zimmerman, & D. H. Schunk (Eds.), Motivation and Self-Regulated Learning: Theory, Research, and Applications (pp. 111-140). New York: Erlbaum.
Perry, N. E., Hutchinson, L. R., Yee, N., & Maatta, E. (2018). Advances in understanding young children’s self-regulation of learning. In D. H. Schunk & J. A. Greeene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning suestionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813. https://doi.org/10.1177/0013164493053003024
Pintrich, P. R., & Zusho, A. (2007). Students’ motivation and self-regulated learning in the college classroom. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence based perspective (pp. 731–810). New York: Springer.
Schellings, G., & van Hout-Wolters, B. (2011). Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 6(2), 83–90. https://doi.org/10.1007/s11409-011-9081-9
Schunk, D. H., & Zimmerman, B. J. (1998). Self-regulated learning: From teaching to self-reflective practice. Psychological Science. Guilford Press.
Schwarz, N., & Oyserman, D. (2001). Asking questions about behaviour. American Journal of Evaluation, 22(2), 127–160. https://doi.org/10.1177/109821400102200202
Shukla, K., & Konold, T. (2018). A two-step latent profile method for identifying invalid respondents in self-reported survey data. Journal of Experimental Education, pp. 1–16. https://doi.org/10.1080/00220973.2017.1315713
Tuckman, B. W. (1991). The development and concurrent validity of the procrastination scale. Educational and Psychological Measurement, 51(2), 473–480. https://doi.org/10.1177/0013164491512022
Usher, E., & Schunk, D. H. (2018). Social cognitive thoretical perspective of self-regulation. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.
Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah, NJ: Erlbaum Associates.
Winne, P. H., & Jamieson-Noel, D. (2003). Self-regulating studying by objectives for learning: Students’ reports compared to a model. Contemporary Educational Psychology, 28(3), 259–276. https://doi.org/10.1016/S0361-476X(02)00041-3
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In Handbook of Self-Regulation (pp. 531–566). Elsevier. https://doi.org/10.1016/B978-012109890-2/50045-7
Wolters, C. A., & Benzon, M. B. (2013). Assessing and predicting college students use of strategies for the self-regulation of motivation. Journal of Experimental Education, 81(2), 199–221. https://doi.org/10.1080/00220973.2012.699901
Wolters, C. A., & Won, S. (2018). Validity and the use of self-report questionnaires to assess self-regulated learning. In D. H. Schunk & J. A. Greeene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). New York: Routledge.
Zimmerman, B. J. (2008). Investigating self-Regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166–183. https://doi.org/10.3102/0002831207312909