It is all in the surv-eye: can eye tracking data shed light on the internal consistency in self-report questionnaires on cognitive processing strategies?
Main Article Content
Abstract
Although self-report questionnaires are widely used, researchers debate whether responses to these types of questionnaires are valid representations of the respondent’s actual thoughts and beliefs. In order to provide more insight into the quality of questionnaire data, we aimed to gain an understanding of the processes that impact the completion of self-report questionnaires. To this end, we explored the process of completing a questionnaire by monitoring the eye tracking data of 70 students in higher education. Specifically, we examined the relation between eye movement measurements and the level of internal consistency demonstrated in the responses to the questionnaire. The results indicated that respondents who look longer at an item do not necessarily have more consistent answering behaviour than respondents with shorter processing times. Our findings indicate that eye tracking serves as a promising tool to gain more insight into the process of completing self-report questionnaires.
Article Details
FLR adopts the Attribution-NonCommercial-NoDerivs Creative Common License (BY-NC-ND). That is, Copyright for articles published in this journal is retained by the authors with, however, first publication rights granted to the journal. By virtue of their appearance in this open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.
References
Ariasi, N., Hyönä, J., Kaakinen, J., & Mason, L. (2017). An eye-movement analysis of the refutation effect in reading science text. Journal of Computer Assisted Learning, 33(3), 202-221. https://doi.org/10.1111/jcal.12151
Ariasi, N., & Mason, L. (2011). Uncovering the effect of text structure in learning from a science text: An eye-tracking study. Instructional science, 39(5), 581-601. https://doi.org/10.1007/s11251-010-9142-5
Baayen, R. (2008). Analyzing linguistic data: A practical introduction to statistics using R. Cambridge: Cambridge University Press.
Baayen, R., Davidson, D., & Bates, D. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of memory and language, 59(4), 390-412. https://doi.org/10.1016/j.jml.2007.12.005
Baddeley, A., & Hitch, G. (1974). Working Memory. In G. H. Bower (Ed.), Psychology of Learning and Motivation (Vol. 8, pp. 47-89): Academic Press.
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01
Beatty, P., & Willis, G. (2007). Research Synthesis: The Practice of Cognitive Interviewing. Public Opinion Quarterly, 71(2), 287-311. https://doi.org/10.1093/poq/nfm006
Catrysse, L., Gijbels, D., & Donche, V. (2018). It is not only about the depth of processing: What if eye am not interested in the text? Learning and Instruction, 58, 284-294. https://doi.org/10.1016/j.learninstruc.2018.07.009
Catrysse, L., Gijbels, D., Donche, V., De Maeyer, S., Van den Bossche, P., & Gommers, L. (2016). Mapping processing strategies in learning from expository text: an exploratory eye tracking study followed by a cued recall. Frontline learning research, 4(1), 1-16. https://doi.org/10.14786/flr.v4i1.192
Collins, D. (2003). Pretesting survey instruments: an overview of cognitive methods. Quality of life research, 12(3), 229-238. https://doi.org/10.1023/A:1023254226592
Conway, A., Kane, M., Bunting, M., Hambrick, D., Wilhelm, O., & Engle, R. (2005). Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review, 12(5), 769-786. https://doi.org/10.3758/bf03196772
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104. https://doi.org/10.1037/0021-9010.78.1.98
Delignette-Muller, M. L., & Dutang, C. (2015). fitdistrplus: An R package for fitting distributions. Journal of Statistical Software, 64(4), 1-34. https://doi.org/10.18637/jss.v064.i04
Desimone, L., & Le Floch, K. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational evaluation policy analysis, 26(1), 1-22. https://doi.org/10.3102/01623737026001001
Dinsmore, D., & Alexander, P. (2012). A Critical Discussion of Deep and Surface Processing: What It Means, How It Is Measured, the Role of Context, and Model Specification. Educational psychology review, 24(4), 499-567. https://doi.org/10.1007/s10648-012-9198-7
Donche, V., & Van Petegem, P. (2008). The validity and reliability of the short inventory of learning patterns. In E. Cools, H. van den Broeck, & T. Redmond (Eds.), Style and cultural differences: how can organisations, regions and countries take advantage of style differences (pp. 49-59). Ghent: Vlerick Leuven Ghent Management School.
Duchowski, A. (2007). Eye tracking methodology: Theory and practice. London: Springer.
Fazio, R. (1990). Multiple Processes by which Attitudes Guide Behavior: The Mode Model as an Integrative Framework. In M. P. Zanna (Ed.), Advances in Experimental Social Psychology (Vol. 23, pp. 75-109). New York: Academic Press.
Fowler, F. (2014). Survey research methods - 5th edition. Thousand Oaks: Sage publications.
Galesic, M., Tourangeau, R., Couper, M., & Conrad, F. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5), 892-913. https://doi.org/10.1093/poq/nfn059
Gathercole, S., & Alloway , T. (2013). De invloed van het werkgeheugen op het leren: Handelingsgerichte adviezen voor het basisonderwijs. Amsterdam: SWP, Amsterdam.
Graesser, A., Cai, Z., Louwerse, M., & Daniel, F. (2006). Question Understanding Aid (QUAID) a web facility that tests question comprehensibility. Public Opinion Quarterly, 70(1), 3-22. https://doi.org/10.1093/poq/nfj012
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.
Hyönä, J., & Lorch, R. (2004). Effects of topic headings on text processing: Evidence from adult readers' eye fixation patterns. Learning and Instruction, 14(2), 131-152. https://doi.org/10.1016/j.learninstruc.2004.01.001
Hyönä, J., Lorch, R., & Kaakinen, J. (2002). Individual differences in reading to summarize expository text: Evidence from eye fixation patterns. Journal of Educational Psychology, 94(1), 44-55. https://doi.org/10.1037//0022-0663.94.1.44
Hyönä, J., Lorch, R., & Rinck, M. (2003). Eye Movement Measures to Study Global Text Processing. In R. Hyönä (Ed.), The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research (pp. 313-334). Amsterdam: Elsevier Science.
Jarodzka, H., & Brand-Gruwel, S. (2017). Tracking the reading eye: towards a model of real-world reading. Journal of Computer Assisted Learning, 33(3), 193-201. https://doi.org/10.1111/jcal.12189
Jobe, J., & Herrmann, D. (1996). Implications of models of survey cognition for memory theory. Basic applied memory research, 2, 193-205. https://doi.org/10.1023/A:1023279029852
Just, M., & Carpenter, P. (1980). A theory of reading: From eye fixations to comprehension. Psychological review, 87(4), 329. https://doi.org/10.1037/0033-295X.87.4.329
Karabenick, S., Woolley, M., Friedel, J., Ammon, B., Blazevski, J., Bonney, C., . . . Kempler, T. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139-151. https://doi.org/10.1080/00461520701416231
Kinnunen, R., & Vauras, M. (1995). Comprehension monitoring and the level of comprehension in high- and low-achieving primary school children's reading. Learning and Instruction, 5(2), 143-165. https://doi.org/10.1016/0959-4752(95)00009-R
Krosnick, J. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236. https://doi.org/10.1002/acp.2350050305
Krosnick, J., & Alwin, D. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201-219. https://doi.org/10.1086/269029
Lenzner, T., Kaczmirek, L., & Galesic, M. (2011). Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research, 23(3), 361-373. https://doi.org/10.1093/ijpor/edq053
Lenzner, T., Kaczmirek, L., & Galesic, M. (2014). Left Feels Right: A Usability Study on the Position of Answer Boxes in Web Surveys. 32(6), 743-764. https://doi.org/10.1177/0894439313517532
Lenzner, T., Kaczmirek, L., & Lenzner, A. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24(7), 1003-1020. https://doi.org/10.1002/acp.1602
Lo, S., & Andrews, S. (2015). To transform or not to transform: Using Generalized Linear Mixed Models to analyse reaction time data. Frontiers in Psychology, 6, 1171. https://doi.org/10.3389/fpsyg.2015.01171
Marsden, P., & Wright, J. (2010). Handbook of survey research - 2nd edition. Bingley: Emerald Group Publishing.
Menold, N., Kaczmirek, L., Lenzner, T., & Neusar, A. (2014). How do respondents attend to verbal labels in rating scales? Field Methods, 26(1), 21-39. https://doi.org/10.1177/1525822X13508270
Neuert, C. (2016). Eye tracking in questionnaire pretesting.
Olsen, A. (2012). The Tobii I-VT fixation filter.
Olsson, P. (2007). Real-time and offline filters for eye tracking.
Pallant, J. (2007). SPSS Survival Manual: A Step by Step Guide to Data Analysis Using SPSS for Windows - 3th edition: Maidenhead: Open University Press.
Penttinen, M., Anto, E., & Mikkilä-Erdmann, M. (2013). Conceptual change, text comprehension and eye movements during reading. Research in Science Education, 43(4), 1407-1434. https://doi.org/10.1007/s11165-012-9313-2
Presser, S., Couper, M., Lessler, J., Martin, E., Martin, J., Rothgeb, J., & Singer, E. (2004). Methods for Testing and Evaluating Survey Questions. Public Opinion Quarterly, 68(1), 109-130. https://doi.org/10.1093/poq/nfh008
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological bulletin, 124(3), 372-422. https://doi.org/10.1037/0033-2909.124.3.372
Rayner, K., Chace, K., Slattery, T., & Ashby, J. (2006). Eye Movements as Reflections of Comprehension Processes in Reading. Scientific Studies of Reading, 10(3), 241-255. https://doi.org/10.1207/s1532799xssr1003_3
Redline, C. D., & Lankford, C. (2001). Eye-movement analysis: a new tool for evaluating the design of visually administered instruments (paper and web). Proceedings of the Survey Research Methods Section of the American Statistical Association.
Richardson, J. (2004). Methodological Issues in Questionnaire-Based Research on Student Learning in Higher Education. Educational psychology review, 16(4), 347-358. https://doi.org/10.1007/s10648-004-0004-z
Richardson, J. (2013). Research issues in evaluating learning pattern development in higher education. Studies in Educational Evaluation, 39(1), 66-70. https://doi.org/10.1016/j.stueduc.2012.11.003
Rossi, P., Wright, J., & Anderson, A. (1983). Handbook of survey research. Sample surveys: History, current practice, and future prospects. San Diego: Academic Press.
Schellings, G. (2011). Applying learning strategy questionnaires: problems and possibilities. Metacognition and Learning, 6(2), 91-109. https://doi.org/10.1007/s11409-011-9069-5
Schellings, G., & Van Hout-Wolters, B. (2011). Measuring strategy use with self-report instruments: theoretical and empirical considerations. Metacognition and Learning, 6(2), 83-90. https://doi.org/10.1007/s11409-011-9081-9
Schwarz, N. (1990). Assessing frequency reports of mundane behaviors: Contributions of cognitive psychology to questionnaire construction. In Research methods in personality and social psychology. (pp. 98-119). Thousand Oaks, CA, US: Sage Publications, Inc.
Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21(2), 277-287. https://doi.org/10.1002/acp.1340
Singleton, R., & Straits, B. (2009). Approaches to social research - 5th edition. Oxford: Oxford University Press.
Staub, A., & Rayner, K. (2007). Eye movements and on-line comprehension processes. In G. Gaskell (Ed.), The Oxford handbook of psycholinguistics: Oxford University Press.
Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73-100). Washington, DC: National Academy Press.
Tourangeau, R., Rips, L., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
Unsworth, N., Heitz, R., Schrock, J., & Engle, R. (2005). An automated version of the operation span task. Behavior Research Methods, 37(3), 498-505. https://doi.org/10.3758/bf03192720
van Gog, T., & Jarodzka, H. (2013). Eye Tracking as a Tool to Study and Enhance Cognitive and Metacognitive Processes in Computer-Based Learning Environments. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 143-156). New York: Springer.
Veenman, M. (2011). Alternative assessment of strategy use with self-report instruments: a discussion. Metacognition and Learning, 6(2), 205-211. https://doi.org/10.1007/s11409-011-9080-x
Veenman, M., & van Hout-Wolters, B. (2005). The assessment of metacognitive skills: What can be learned from multi-method designs? In C. Artelt & B. Moschner (Eds.), Lernstrategien und Metakognition: Implikationen für Forschung und Praxis (pp. 77-99): Münster: Waxmann.
Vermunt, J., & Donche, V. (2017). A learning patterns perspective on student learning in higher education: state of the art and moving forward. Educational psychology review, 29(2), 269-299. https://doi.org/10.1007/s10648-017-9414-6
Willis, G., & Miller, K. (2011). Cross-cultural cognitive interviewing: Seeking comparability and enhancing understanding. Field Methods, 23(4), 331-341. https://doi.org/10.1177/1525822X11416092
Willis, G., Royston, P., & Bercini, D. (1991). The use of verbal report methods in the development and testing of survey questionnaires. Applied Cognitive Psychology, 5(3), 251-267. https://doi.org/10.1002/acp.2350050307
Yeari, M., Oudega, M., & van den Broek, P. (2016). The effect of highlighting on processing and memory of central and peripheral text information: evidence from eye movements. Journal of Research in Reading, 40(4), 365-383. https://doi.org/10.1111/1467-9817.12072