Does Personalisation Promote Learners’ Attention?                           
An Eye-Tracking Study

Steffi Zander, Maria Reichelt, Stefanie Wetzel, Frauke Kämmerer, Sven Bertel

Bauhaus-Universität Weimar, Germany

 

                                                                                                               Article received 29 Marc 2015 / revised 6 May 2015 / accepted 24 June 2015 / available online 5 November 2015

Abstract

The personalisation principle is a design recommendation and states that multimedia presentations using personalised language promote learning better than those using formal language (e.g., using ‘your’ instead of ‘the’). It is often assumed that this design recommendation affects motivation and therefore allocation of attention. To gain further insight into the processes underlying personalisation effects we conducted an eye tracking experiment with 37 German university students who were presented with either personalised or formal learning materials. We examined group differences in attention allocation parameters (fixation rate, mean fixation duration, transition count, reading depth). The eye-tracking data was combined with self-reports concerning motivation, cognitive load, and learning outcomes. Eye-tracking data revealed a higher reading depth for the main picture areas of interest in the personalised condition. Additionally, participants found the personalised version more appealing and inviting. For learning outcomes, there was a positive effect of personalisation. However, after Bonferroni correction effects and therefore the pattern expected did not reach significance. The results are discussed in regard to their importance for methodological and practical implications for instructional design.

Keywords: multimedia learning; personalisation effect; motivation and learning; eye-tracking; mixed-methods 

Corresponding author: Dr. Steffi Zander, Faculty of Art and Design, Chair of Instructional Design. Geschwister-Scholl-Straße 7, 99423 Weimar. Germany. Phone: +49(0)3643 / 58 32 29, Fax: +49(0)3643 / 58 32 48, Email: steffi.zander@uni-weimar.de DOI: http://dx.doi.org/10.14786/flr.v3i4.161

1.          Personalisation Effects in Multimedia Learning

Learning of multimedia content can be affected by modest changes to the wording of learning materials. For example, learning is promoted by personalising formal texts, that is, by changing ‘the’ to ‘your’, ‘one’ to ‘you’, and then including direct comments to learners. This approach is known as the personalisation principle, which assumes that multimedia presentations using personalised language promote learning better than those that use formal language. Further, based on empirical findings (cf. Moreno & Mayer, 2000, 2004), even modest changes create personalisation effects. Several studies have revealed personalised language effects, mostly for transfer and retention, but also for motivation (interest and intrinsic motivation) and for perceived cognitive load (difficulty and invested mental effort). However, results of existing studies are not consistent with regard to motivation and cognitive load. One reason for this is that the variables underlying personalisation effects do not allow for reasoning based on simple causal chains (Ginns, Martin, & Marsh, 2013). The following chapter reflect the existing theoretical framework of the personalisation principle.

1.1.         Theoretical Framework

Although personalisation effects are well documented, it remains unclear which processes are responsible for the beneficial effects on learning outcomes. These effects have mainly been investigated using subjective measures at the end of learning phases, whereas objective process-oriented measures have largely been overlooked. In the field of multimedia learning, several theoretical approaches have been proposed to explain personalisation effects (Reichelt, Kämmerer, Niegemann, & Zander, 2014). These include social agency theory (Mayer, 2005; Mayer, 2009), the effect of stronger familiarity (Moreno & Mayer, 2000a) and the self-reference effect (Rogers, Kuiper, & Kirker, 1977). Overall, the assumptions within these approaches can be subsumed into two basic underlying processes: (1) facilitation of cognitive processing and (2) focusing of cognitive processing driven by a personalised language style. ‘Facilitation’ and `focusing´ here refer to the notion that personalised messages act as a social cue. In a cognitive view, the cue activates other internal cues that enable learners to more easily connect new information to internal structures of the self, via self-referencing processes (Rogers et al., 1977). In a motivational-emotional view, this social cue causes a feeling of social presence or familiarity (Mayer, 2005, 2009; Mayer & Moreno, 2000).  In turn, these processes result in a higher intrinsic motivation, situational interest and in decreased perceived cognitive load during learning (Moreno & Mayer, 2000); they also facilitate the encoding, organisation, and elaboration of relevant information (Moreno & Mayer, 2000; Rogers et al. 1977), ultimately leading to improved learning outcomes.

In our current study personalisation effects were investigated using an objective, process-oriented measure (eye-tracking analysis) to test whether the positive effects of a personalised language style can be traced back to differences in the allocation of attention resources and therefore to deeper and more focused processing (due to personalised language style). Differences in allocation of attentional resources were measured by eye-tracking parameters including fixation duration, fixation count, and transitions between different sources of information (e.g. text and images).

1.2.         Current Studies

Previous research confirming the personalisation principle shows that people learn better from multimedia presentations when words are presented in a personalised language style rather than a formal style (Mayer, 2009). However, not all studies have demonstrated an effect of personalisation on motivation, cognitive load, and learning outcomes (for an overview see Ginns et al., 2013). Table 1 gives an overview of research into the personalisation principle, listing authors and results for several dependent variables. The studies reported in Table 1 investigated learning outcomes mainly as a result of language style, revealing positive effect of personalised language on transfer performance (except Kurt, 2011) and retention (except Kurt, 2011; Mayer et al., 2004; Reichelt et al., 2014; Schworm & Stiller, 2012). Together, these findings support the concept of focused processing being driven by personalised messages. The other two variables of interest, motivation (interest) and cognitive load (perceived difficulty of the material and subjective mental effort), have been applied only in select studies. Consequently, no empirically proven pattern has been revealed concerning the assumption that these variables facilitate processing.

Furthermore, in their meta-analysis of these inconsistent findings, Ginns et al. (2013) showed a diversity of effects and reported several variables with the potential to moderate the effect of personalised language on motivation, retention, and transfer performance. As a consequence, these authors suggested using more fine-grained methods to analyse the effect. For example, only the study of Reichelt et al. (2014) was a mixed-methods-study combining experiments with think-aloud method to examine the personalization effect (see Table 1, column 3). However, Ginns et al. (2013) as well as Reichelt et al. (2014) emphasize the potential of a multi-methods-design to gain more information on underlying processes why personalization effects occur.

1.3.         Research Gaps

The overview of measurements applied in personalisation studies (Table 1) shows that study results are based mainly on subjective self-report of learners. To date, self-report instruments are the only method that has been used to test the personalisation principle. The application of alternate methods is desirable (Ginns et al., 2013) and a potentially useful approach would be to measure the allocation of attentional resources (using eye-tracking methods) during multimedia presentations as this is seen to reveal information regarding the focusing approach. Indeed, this method has already shown that eye movements are an indicator of depth and/or direction of information processing in multimedia learning, according to manipulations of visual or audio-visual characteristics of the learning material. For example, de Koning, Björn B., Tabbers, Rikers, and Paas (2010) investigated cognitive processing during learning of animations containing visual cues, while Johnson and Mayer (2012) examined the processing of spatially contiguous and non-contiguous textual and pictorial information. These findings, together with those of Moreno & Mayer (2000), suggest that effective processing across text and pictures might also be promoted by using a personalised style (as a social cue), thus promoting focused information processing. Therefore, eye-tracking methodology was applied in the present study to analyse whether personalisation affects the allocation of attentional resources and thus provides support for the personalisation principle.

1.4.         Aims and Research Questions of the Study

The literature review showed that the effect of personalized language on attentional processes were not considered so far. Therefore, to fill the research gaps identified above, our eye-tracking study aimed to investigate (1) the impact of personalised language on motivation, cognitive load and learning outcomes, and their possible relation to (2) the processes of allocation of visual attention resources as an indicator of deeper and more focused information processing (driven by personalised messages). We therefore examined the following research questions: (1) Does personalised learning material promote learning processes better than formal text versions (in terms of motivation, cognitive load, and learning outcomes)? (2) Which attention processes underlie the personalisation effect? Based on the literature review (e.g., Ginns et al., 2013; Kartal, 2010; Mayer et al., 2004), we hypothesised that a personalised language style increases learners’ intrinsic motivation, reduces cognitive load, and improves their learning outcomes (Hypothesis 1). Based on current theoretical models (Reichelt et al., 2014, Keller, 2009), we further assumed that learners who received a personalised version of a multimedia presentation would allocate their attention resources with more focus on the relevant areas of the learning material than would learners who received a formal version. This difference should be reflected in (a) increased fixation counts (as a parameter for task difficulty), increased duration of fixations (as a parameter for the amount of effort to process complicated texts, Rayner & Pollatsek, 1989), and a higher number of transitions between text and pictorial information (as a parameter for the process of connecting and integrating information, Holsanova, Holmberg, & Holmqvist, 2009) in the relevant areas for learners who receive personalised presentations (Hypothesis 2).

Table 1

Overview of results for key personalisation studies

 

author and year of publication	transfer	retention	interest	intrinsic motivation	mental effort	cognitive load	task difficulty	friendliness	helpfulness
Moreno and Mayer (2000a)	+1	+
Moreno and Mayer (2004b)	+	+					+	+	+
Moreno and Mayer (2004b)	+	-2	03				0
Kartal (2010)	+	+	+				+	+
Ginns and Fraser (2010)			0		+
Kurt (2011)	0	0				-
Schworm and Stiller (2012)	+	0
Rey and Steib (2013)	+	+	0
Reichelt et al. (2014)	0	+	+	+

¹ (+) means that the effect of personalization on this variable (e.g., transfer) was positive

² (-) means that the effect of personalization on this variable (e.g., transfer) was negative (in favour of formal texts)

³(0) means that were no differences between formal and personalized condition

2.          The Eye-Tracking Study: Methods and Materials

2.1           Participants and Design

Participants were 37 college students (mean age = 25.03, SD = 3.436; male = 21) at Bauhaus-Universität Weimar and the University of Erfurt in Germany. The participants received either a personalised (n = 19) or a formal (n = 18) version of a computer-based program about typical weather phenomena.

To test the influence of domain specific prior knowledge, we used a Kruskal-Wallis test. The test showed a non-significant result (χ² = 0.001, p = 0.975); therefore, we assumed an equal distribution of prior knowledge in the experimental groups can be assumed.

2.2           Learning Material

The multimedia learning material consisted of a combination of static pictures and on-screen text, presented on seven slides and with a total duration of approximately 10 minutes. In accordance with Mayer (2009), we used various techniques for creating a personalised style. Personalisation of the formal text was achieved by replacing impersonal articles with possessive pronouns and third person constructions with second person constructions. Only the text was personalised. Table 2 shows examples of this manipulation.

 

Table 2

Examples of Personalized and Formal Text Versions

 

Formal Style	Personalized Style
The task is… The picture shows a tropical storm…	Your task is… You can see a picture of a tropical storm…

2.3           Procedure

2.3.1      Measurements

The pre-test phase consisted of a task description in either formal or personalized language style, a questionnaire on learners’ initial motivational state (QCM, dimension situational interest, Rheinberg, Vollmeyer, and Burns, 2001), and a prior knowledge test. After this, the eye-tracker was adjusted and the learning phase began. After completing the learning phase, participants rated (1) how inviting and personally appealing they perceived the language style (for manipulation check), (2) their intrinsic motivation based on the questionnaire by Isen and Reeve (2005), and (3) their perceived cognitive load (as a measure of perceived difficulty, Koch, Seufert, and Brünken, 2008). Ratings were provided on a 7-point-Likert scale (“I disagree” to “I agree”). (4) Following this, participants gave responses on the retention and transfer test (learning outcome).

The investigations were conducted in a computer laboratory at Bauhaus-Universität Weimar. To capture the eye movements of the participants, we used an SR Research EyeLink II head-mounted eye tracker. The participants were placed 55 – 60 cm in front of a 24-inch monitor. Fixations, saccades and blinks were recorded at 250 Hz for the dominant eye of each participant. A linear drift correction (to the screen centre) was implemented after presentation of each stimulus screen. A second calibration was completed before presentation of the learning material (using a 9-point calibration).

2.3.2      Data Analyses

For gaze data analysis, we measured fixation rate, overall fixation duration, and average fixation duration for pre-defined areas of interest on the stimulus screens. To divide the screen into areas of interest (AOIs), we used analysis based on the expected findings regarding personalisation effects (hypotheses based method, see Figure 1).

 

Figure 1. Predefined AOIs based on hypotheses.  (picture modified for eye-tracking analysis based on the following source: http://www2.klett.de/sixcms/media.php/76/karte_desertifikation.jpg). (see pdf)

Moreover, to verify the resulting AOIs cluster analysis (data driven method, see Figure 2) was used. Figure 2 shows an example screenshot for the formal style (right) and the personalised style (left). Both approaches revealed similar partitions of the stimulus screens. However, we decided to analyse our gaze data based on the predefined AOIs (Figure 1) because the granularity level of the cluster analysis was too high.

 

Figure 2. Screenshots of the extracted AOIs developed by cluster analysis. (see pdf)

 

Additionally, we considered fixation transitions between pairs of AOIs (i.e. a fixation on one AOI followed by a fixation on another AOI, irrespective of transition direction). AOI locations and extent were set based on pre-defined hypotheses (e.g. encompassing all text or an entire diagram). Locations and extent were verified for each stimulus screen according to a post-hoc cluster analysis based on the DBSCAN algorithm (with 4 gaze samples minimum per cluster and 35 pixels maximum distance between gaze samples) and by visual inspection of heat maps; less than 5% of fixations were found to lie outside of the AOIs.

In the first analysis step, we examined the data concerning fixations and transitions for the AOIs. To accomplish this we analysed fixations and transitions for the picture and text for both the personalized and formal stimulus groups (Figure 1, aoi_text and aoi_picture). For subsequent steps of the analysis, the main text and picture AOIs were further subdivided into smaller AOIs; this process was hypothesis-driven.

 

3.          Results 

We used Mann-Whitney U tests for the statistical analysis because prior testing (Shapiro-Wilk test) revealed that data were not normally distributed. Effect size is reported using r. Overall we analysed 19 dependent variables (see Table 3 and Table 4) which increases the type I error. Therefore we used Bonferroni correction to calculate a new alpha level. We will describe the results considering both alpha levels, namely the corrected (α = 0.002) and the uncorrected (α = 0.05).

3.1.         Manipulation check

Comparisons of the participant perceptions of the two language styles (formal and personalised) revealed that participants found the personalised presentation more appealing (U = 101, z = -2.207, p = 0.034, r = -0.363) and inviting (U = 82.5, z = -2.780, p = 0.006, r = -0.457) than the formal presentation. However, the results are not statistically significant after Bonferroni correction.

3.2.         Hypothesis 1: Personalized Language, Motivation, and Learning Outcomes

Table 3 provides an overview of medians for each variable test in Hypothesis 1. Although, the descriptive data show a trend towards the expected effect in favour of the personalised version, the results for the motivational variables showed no significant difference between learners who viewed a personalised text compared with those who viewed a formal version. This non-significant effect was found for both situational interest (U = 138, z = -1.008, p = 0.313, r = -0.166) and intrinsic motivation (U = 142, z = -0.883, p = 0.377, r = -0.145).

Further, the descriptive analysis confirms the assumption that learners who viewed personalised learning material estimated their cognitive load to be lower than did those who learned with a formal version. However, there were also no significant differences between ratings for learners’ cognitive load (U = 150.5, z = -0.625, p = 0.532, r = -0.103) for formal and personalised presentations.

The trend in descriptive data was also reflected in the learning outcome variables. There were differences (α = 0.1, uncorrected) in learners’ retention of personalised vs. formal presentation materials (U = 119, z = -1.654, p = 0.098, r = -0.271), such that participants who viewed a personalised computer-based program showed superior retention compared to those who viewed a formal version. Again, after applying Bonferroni correction the results are not significant. For the transfer test, the differences were not significant (U = 168.5, z = -0.082, p = 0.935, r = -0.013).

 

Table 3

Medians for all variables for formal and personalised conditions (hypothesis 1)

 

Variables and Measures	Formal	Personalised
Manipulation Check
Perceived personal appeal of language style       Perceived inviting character of language style	15.11 14.08	22.68 23.66
Motivation      Initial motivation (before)      Intrinsic motivation (after)	17.17 17.39	20.74 20.53
Cognitive Load (based on perceived difficulty)	20.14	17.92
Learning Outcome       Retention       Transfer	16.11 18.86	21.74 19.13

3.3.         Hypothesis 2: Eye-Tracking Analysis

Table 4 shows all median data for gaze analysis for both personalised and formal presentation styles. In the first instance the results over all screens are reported. Afterwards, data for single screen 3 are presented to accentuate the findings on a more fine-grained level.

The eye-tracking analyses revealed several group differences (formal vs. personalisation) in fixation rate and reading depth. The fixation rate was higher in the personalised condition than in the formal condition whereas average fixation duration on the main text AOIs was greater in the formal condition than in the personalised condition. Lower fixation rates indicate greater task difficulty (Minoru Nakayama, Koji Takahashi, & Yasutaka Shimizu, 2002) while greater average fixation duration indicates more effortful cognitive processing (Rayner & Pollatsek, 1989) that is necessary for more complicated texts. Thus, these values indicate that the personalized text was easier to understand and more easily processed by the learners. However, none of the observed differences reached significance, neither using the uncorrected alpha level of 0.05 (there were marginal differences at the 10% level), nor the corrected alpha level of 0.002. For the picture aspect of the stimuli, results were that participants demonstrated greater reading depth for the main picture AOIs in the personalised condition than in the formal condition. Reading depth is defined as the accumulated time spent looking at the AOI divided by the AOI area in cm². This measure indicates how much of the text has been read or how much of a picture has been examined (Holmqvist et al., 2011). The higher value in the personalised condition suggests more intensive observation of the picture than in the formal condition. This is supported by the descriptive finding that the number of transitions between text and picture AOIs tends to be greater for the personalised learning material. A greater number of transitions between AOIs with semantic relations indicates better connection and integration of the presented information (Holsanova et al., 2009).

The assumptions are moreover fostered by combining these results with data on retention. Therefore, each screen analysis was inspected to verify the findings. Comparisons of personalised vs. formal presentations for the individual screens in corresponding pairs revealed similar patterns of differences for the fixation rate on legends, average fixation durations on text AOIs, reading depth on pictures, and transitions between several map components. Many of these differences were statistically significant at the 0.1 alpha level but not at the corrected 0.002 alpha level. For example, reading depth for the picture on screen 3 (Figure 2) was higher for personalised than for formal learning material. Further, the average fixation duration on text AOIs differed between personalised and formal presentations. Both differences reached significance at the 10% level. The situation is again comparable when comparing AOIs created by subdividing the main picture AOI into legend and picture proper. In this case, there were greater numbers of transitions between the legend and picture proper and a higher fixation rate on the legend in the personalised condition than in the formal condition. Results for the individual screen 3 support the findings for the combined screen analysis, indicating that learners who viewed the personalised learning materials paid more attention to the pictorial material than did those who viewed the formal learning materials. This finding is supported by the retention results and by the finding that learning time for screen 3 differed between formal and personalised presentations, with greater learning time for the personalised presentation on a descriptive level. As an additional measure this finding indicates a deeper processing of information under the personalised condition. Those results have to be interpreted and discussed with caution.

Table 4

Medians and statistical results for gaze data for two conditions, formal and personalised (hypothesis 2)

 

Gaze data parameters	Formal (Median)	Personalised (Median)	U	z	p	r
Gaze Data (All Screens Combined)
Fixation rate	3.73	4.03	123	-1.66	0.099	-0.27
Average fixation duration (ms) on text AOI	176.01	158.44	123	-1.66	0.099	-0.27
Average fixation duration (ms) on picture AOI	44.53	42.38	163	-0.497	0.633	-0.08
Reading depth (s/cm2) on text AOI	150.72	156.67	160	-0.585	0.573	-0.09
Reading depth (s/cm2) on picture AOI	24.11	31.37	109	-2.076	0.038	-0.34
Transitions (per s) between text and picture AOI	0.102	0.118	148	-0.936	0.361	-0.15
Gaze Data (Screen 3)
Reading depth (s/cm2) on text AOI	135.376	141.820	138	-1.228	0.228	-0.20
Reading depth (s/cm2) on picture AOI	25.98	38.68	122	-1.696	0.093	-0.28
Average fixation duration (ms) on text AOI	178.90	168.00	121	-1.725	0.087	-0.28
Average fixation duration (ms) on picture AOI	46.08	46.86	151	-0.848	0.409	-0.14
Fixation rate on legend (aoi_legend)	0.23	0.27	119	-1.783	0.077	-0.29
Transitions between legend and picture proper (aoi_legend & aoi_map)	0.04	0.09	100.5	-2.324	0.019	-0.38

4.          Discussion

The presented study aimed to investigate whether the effect of personalised language style on learning outcomes can be associated with motivational and cognitive load issues and differences in the pattern of attention allocation based on gaze pattern analyses.

On a descriptive level, our findings confirm the assumption that personalisation affects learners’ motivation and their perceived cognitive load; however, the results were non-significant. For learning outcomes, there was a positive effect of personalisation for retention but not for transfer. These somewhat inconsistent findings are in accordance with previous studies (e.g. Ginns et al., 2013). The inconsistency in findings makes it necessary to further investigate underlying processes. Therefore, we conducted more detailed examinations of the allocation of attention resources and other explanatory variables. For the former, the eye-tracking data show the expected pattern of results with a greater number of transitions between main AOIs in the textual and pictorial information, along with higher fixation duration on the main picture AOIs. These results provide evidence that Mayer’s (2009) proposition that people engage in more focused processing of personalised learning material than of formal material. Unexpectedly, in our study, this result was found only for pictorial information but did not extend to textual information. Learners of personalised material may pay attention not only to the text but also to the picture, which in turn is reflected in a higher transition count between text and picture and higher fixation count and duration. This assertion is supported by the data for individual screen 3, which revealed a difference with regard to learning time spent on the screen, with longer learning time for the personalised presentation than for the formal presentation. Why do these findings occur and what limitations should be considered when interpreting the results?

Overall, the results of our study confirm that the combination of methods was a fruitful approach for clarifying a very complex set of interacting variables. To add to this approach, we suggest that, in future studies, gaze data should be combined with retrospective interviews (Van Gog et al. 2005, cued retrospective reporting) while learners view the gaze distributions. This would make it possible to obtain more fine-grained information concerning motivation and cognitive load from reflections about the learning process. In the same vein, measuring learning outcomes after each screen presentation should provide a better match between gaze behaviour and learning results. Future research could also include the analysis of the sample for any relevant differences, particularly with regard to their educational disciplines. Such differences may act as moderator variables (Ginns et al., 2013; Reichelt et al., 2014).

As main limitation, the sample size should be discussed. The sample was small, suggesting a lack of explanatory power, especially with regard to data on learning outcomes, motivation and cognitive load. However, a small sample size is typically for eye-tracking studies (Goldberg & Wichansky, 2003) and it is justified by the individual surveys. To give generalized statements regarding motivation and learning, a larger sample size is needed. Hence, our investigation should be replicated with more participants to increase the power.

Although our findings suggest support for personalisation theory, the data should be interpreted with caution. In particular, the data on learning outcomes, especially for transfer, frequently did not reach statistically significant levels. Several limitations may be responsible for these inconsistencies in our results. To begin with, because our hypotheses included learning material as a whole, we did not focus on pictorial information. To measure learning outcomes with regard to the pictorial information would have required the implementation of explicit pictorial tasks. Future studies should contain more detailed pictorial analyses.

Moreover, with regard to procedures and physical aspects of the study, our eye tracker was head mounted and had to be re-calibrated after every screen presentation. Both of these circumstances may have affected the availability of attention resources for learning and understanding. For example, participants had to concentrate on sitting still and were subject to interruptions of the learning process during the re-calibrations. Future studies should apply less intrusive methods of recording gaze data.

Another possible limitation is linked to the time required for learning. One important and unanswered question is whether the personalisation effect (for fixation, transition, learning outcome) can vanish if the learning time (duration of presentation) increases. This question should be tested in future studies to determine the practical implications for instructional design and especially for the improvement of design principles in multimedia learning environments.

Keypoints

       Eye-tracking measures can be applied to study the effects of personalisation of learning material on learning outcomes.

       The combination of eye movement data and self-report reveals that personalised learning material may be processed more deeply than formal material.

       Eye-tracking data suggest that people engage in more focused processing of personalised learning material than formal learning material.