The complex relationship between students’ critical thinking and epistemological beliefs in the context of problem solving

Heidi Hyytinen^a, Katariina Holma^b, Auli Toom^a, Richard J. Shavelson^c, Sari Lindblom-Ylänne^a

^a University of Helsinki, Finland

^b University of Eastern Finland, Finland

^c SK Partners, LLC & Graduate School of Education, Stanford University, USA

Article received 2 May 2014 / revised 14 June 2014 / accepted 27 July 2014 / available online 24 September 2014

Abstract

The study utilized a multi-method approach to explore the connection between critical thinking and epistemological beliefs in a specific problem-solving situation. Data drawn from a sample of ten third-year bioscience students were collected using a combination of a cognitive lab and a performance task from the Collegiate Learning Assessment (CLA). The cognitive-lab data were analysed using thematic analysis. The findings showed that students’ epistemological beliefs were interwoven into their critical thinking: students used critical thinking as a tool (1) for enhancing understanding and (2) for determining truth or falsehood. Based on this classification, students could be placed in one of two qualitative profiles, either (1) thorough processing or (2) superficial processing. The results indicated that students who showed superficial processing palmed off justification for knowing on authoritative figures. In contrast to previous studies these students did not consider knowledge to be absolutely certain or unquestionable. The findings also show that students with thorough processing believed knowledge to be tentative and fallible, but did not share the relativist view of knowledge where any claim counts because all knowledge is relative. All ten students shared a fallibilist view of knowledge.

Keywords: Critical Thinking; Epistemological Beliefs; Cognitive Lab; Relativism; Fallibilism

Corresponding author: Heidi Hyytinen, Institute of Behavioural Sciences, the University of Helsinki, 00014 University of Helsinki, Finland, Email: heidi.m.hyytinen@helsinki.fi

Doi: http://dx.doi.org/10.14786/flr.v2i4.124

1.                  Introduction

Critical thinking has been singled out as one of the most important skills for citizens of the twenty-first century (Halpern, 2014). Mastering critical thinking is thus a goal that can be found in almost every higher education curriculum today. However, recent studies have raised concerns that even though most students make significant progress in learning concepts and procedures during their university studies, some students show little if any growth in critical thinking (Arum & Roksa, 2011a, 2011b; Bok, 2006; Pascarella, Blaich, Martin & Hanson, 2011).

In the field of higher education, research on critical thinking has generally focused on the development of critical thinking skills (e.g. Arum & Roksa, 2011a; Heijltjes, van Gog, Leppink & Paas, 2014). Researchers have also highlighted the importance of understanding critical thinking as a social activity (e.g. Arum & Roksa, 2011b; Kuhn, 2005; Moore, 2004; 2013). In this exploratory study we provide a multidimensional framework for analysing critical thinking by combining theoretical aspects from philosophical, educational and psychological approaches. In our view the concept of critical thinking is closely connected to the concepts of ‘knowledge’ and ‘knowing’. Furthermore, we assume that critical thinking cannot be formulated by referring to skills alone, but also always involves a disposition to use these skills adequately (see Bailin & Siegel, 2003; Holma, 2014; Siegel, 1988).

Previous research on critical thinking and personal epistemology has frequently applied quantitative multiple-choice tests, questionnaires or qualitative interviews (see e.g. Australian Council of Education Research, 2001; Heijltjes, van Gog, Leppink & Paas, 2014; Greene & Yu 2014; Lahtinen & Pehkonen, 2013; Tremblay, Lalancette & Roseveare, 2012). Recently, many researchers have questioned the reliability and adequacy of self-report questionnaires (Greene & Yu, 2014; Elby & Hammer, 2001). As a result, researchers have stated that there is a need for studies that assess the performance of students directly (e.g. Elby & Hammer, 2001; Hofer, 2004; Stes, Min-Leliveld, Gijbels and van Petegem 2009). At the same time researchers have also assumed that one assessment method is not enough to evaluate complex cognitive processes such as reasoning (e.g. Baartman, Bastiaens, Kirschner & Vleuten, 2007; Dierick & Dochy, 2001; Maclellan, 2004). This study responds to current concerns by exploring students’ critical thinking as well as their epistemological beliefs, as elaborated upon below, in a problem-solving situation to which we applied a multi-method qualitative approach.  A think-aloud method was used as the students worked through an open-ended performance task. Our aim is to identify and understand qualitative differences in the critical thinking of students and in their beliefs about knowledge, as well as in their personal relationships.

2.         Critical thinking in university-level studies

Critical thinking is often ‘regarded as fundamental aim of education’ (Bailin & Siegel, 2003, p.188; cf. Dewey, 1910). In a university context critical thinking has an essential role and is an important component of the learning outcomes (Bok, 2006). Critical thinking is defined as a process that enables an individual to make an informed decision about conflicting claims (Ennis, 1991; Fisher, 2011; Bailin & Siegel, 2003). It is purposeful, reasoned and reflective thinking (Ennis, 1991; American Philosophical Association, 1990). A critical thinker knows how to assess the strength of evidence and the reasons that are relevant to the particular context or type of task, and also shows the disposition to draw on these skills (Bailin & Siegel, 2003; Scheffler, 1965, Halpern, 2014).

Critical thinking is seen as a skilful activity in which a person may be more or less proficient (Fisher, 2011; Scheffler, 1965). Definitions of critical thinking typically include a list of the thinking skills that characterise an ideal critical thinker. For example, Fisher (2011) lists the following: the ability to identify the elements in a reasoned case, especially reasons and conclusions; the abilities to identify and evaluate assumptions; the abilities to clarify and interpret expressions and ideas; to be able to judge the acceptability, especially the credibility, of claims; to evaluate arguments, analyse, evaluate and produce explanations; to be able to analyse, evaluate, and make decisions; to draw inferences and produce arguments (see also Halpern, 2014). University studies require all of these abilities.

However, many philosophers have argued that critical thinking cannot be conceptualised merely by referring to a prescribed set of skills (Bailin & Siegel, 2003; Holma, 2014; Fisher, 2011; Siegel, 1988, Scheffler, 1965; see also Halpern, 2014). It may be that a person has acquired the skills, but does not use them (Fisher, 2011). As Holma (2014) has pointed out, it is not enough for students to have critical thinking skills; they also need to use these skills effectively. Thus, critical thinking always involves both the essential skills or abilities and the disposition to use them (Bailin & Siegel, 2003, Holma, 2014; Siegel, 1988).

Previous studies have called attention to the fact that students’ critical thinking skills do not always develop during university studies (Arum & Roksa, 2011a; Bok, 2006; Pascarella, Blaich, Martin & Hanson, 2011). Arum and Roksa (2011b) demonstrated in their longitudinal study that a large number of university students showed no significant improvement in a range of critical thinking skills, such as reasoning and problem solving. However, a recent study by Heijltjes and colleagues (2014) has shown that the combination of explicit instruction and practice has proven successful in improving students’ performance in reasoning skills.

3.         Knowledge and knowing in critical thinking

Critical thinking demands a comprehensive use of different types of knowledge (Bok, 2006; Ennis, 1991). There is a reciprocal relationship between ‘critical thinking’, ‘knowledge’ and ‘knowing’; on the one hand, students need knowledge about a phenomenon before they can think about it critically (Halpern, 2014); on the other hand, students must have the necessary skills to evaluate that knowledge. The concepts of ‘knowledge’ and ‘knowing’ are thus substantial aspects of conceptualising critical thinking. 

There are several different definitions and classifications of the concept of knowledge. For example, philosophical epistemologists usually differentiate amongst three types of knowledge: propositional knowledge, procedural knowledge and knowledge by acquaintance (Everitt & Fisher, 1995; Ichikawa & Steup, 2012), although there is no consensus on the interpretation of knowledge or on the number of types of knowledge (Fenstermacher, 1994). For our purposes the distinction between propositional and procedural knowledge has theoretical importance.

Propositional knowledge is defined as knowing that ‘such-and-such is the case’. This is sometimes referred to as factual or declarative knowledge. Propositional knowledge (i.e. ‘knowing that’) is usually distinguished from procedural knowledge (i.e. ‘knowing how’) (Ryle, 1949). In philosophical discussions propositional knowledge is related to such epistemological concepts as truth, justification, reason and evidence (Ryle, 1949; Scheffler, 1965, see also Niiniluoto, 1999; Shope, 2004). Scheffler (1965) argued that the ‘knowing that’ attributes of a person may reveal his epistemological orientations, such as the criteria for justifying knowing. Empirical research on personal epistemology focuses particularly on these personal orientations. 

Procedural knowledge, meaning ‘knowing how’ to do something (knowing how to analyse, knowing how to swim, etc.; see Everitt & Fisher, 1995; Shope, 2004), is related to possessing a skill (Scheffler, 1965). In this sense critical thinking represents procedural knowledge, which is consistent with the other aspect of critical thinking mentioned above. However, several researchers have assumed that procedural knowledge always involves some propositional knowledge (i.e. Everitt & Fisher, 1995; Smith 2002; Markowitsch & Messerer, 2007). For example, if a person knows how to play chess, he will probably know certain facts (e.g. rules) about playing chess. Smith (2002) has emphasized that an individual has a certain skill only when his performance reflects both procedural and propositional knowledge. 

In sum, critical thinking involves a disposition to think critically, having the necessary propositional knowledge about a phenomenon and having the thinking skills (i.e. procedural knowledge) to evaluate that knowledge (cf. Halpern, 2014).

The term ‘personal epistemology’ or, alternatively, ‘epistemological belief’ is defined as an individual’s views of the nature of knowledge and knowing. The term also includes a view of one’s personal beliefs as a knower (Pintrich, 2002; Hofer, 2004). The concept of ‘personal epistemology can be described along a continuum from less sophisticated to more sophisticated’ ways of knowing (Kaartinen-Koutaniemi & Lindblom-Ylänne, 2012, p. 2) or a progress ‘from a state of simple, absolute certainty into a multifaceted, evaluative system’ (West, 2004, p. 61). During this process the individual changes from a passive recipient of knowledge to an active participant in constructing and evaluating knowledge (Hofer & Pintrich, 2002; Kuhn, 2005; King & Kitchener, 2004). Over time epistemological beliefs develop more and more toward relativistic beliefs (Hofer & Pintrich, 1997, 2002).

Previous research on personal epistemology has found that the ability to think critically is embedded in a progression of epistemological beliefs (i.e. King & Kitchener, 2004; Kuhn & Weinstock, 2002; Kuhn, 1999; 2005). Several researchers have hypothesised that students with weak critical thinking skills have an absolute view of knowledge. When students move on to the most developed epistemological level, their critical thinking tends to improve as well (Bok, 2006; Kuhn, 1999; Kuhn & Weinstock, 2002).  It has also been demonstrated that students’ epistemological beliefs play an important role in their ability to evaluate the credibility of competing claims (Barzilai & Zohar, 2012).

Whether instruction has any influence on the development of epistemological beliefs is currently under discussion (e.g. Valanides & Angeli, 2005; Lahtinen & Pehkonen, 2013). However, there is evidence that not all university students reach the most highly developed level of personal epistemology (Kuhn & Weinstock, 2002; Kaartinen-Koutaniemi & Lindblom-Ylänne, 2012; King & Kitchener, 2004; Perry, 1970). King and Kitchener (2004) have found that only advanced doctoral students consistently show the highest level of epistemological beliefs. Furthermore, Kaartinen-Koutaniemi and Lindblom-Ylänne (2008, 2012) have shown that there is a considerable variation in personal epistemology among final-year master’s students. Their results also showed variations between students in different age groups, study phases and disciplines (see also Hofer, 2006; Muis, Bendixen & Haerle, 2006). In addition, researchers have assumed that students’ epistemological beliefs may vary within the same discipline or domain (Hammer & Elby, 2003; Greene & Yu, 2014). 

5.         Critical thinking and different conceptions of knowledge

As the brief review above indicates, the literature of personal epistemology makes a distinction between a lower level of epistemological beliefs, in which knowledge is perceived as consisting of unchanging facts and is acquired directly from external authorities, and higher level epistemological beliefs, in which knowledge is seen as uncertain and constructed by the individual himself (Kuhn & Weinstock, 2002; Hofer, 2005; Valanides & Angeli, 2005). Several researchers have stated that students with higher-level epistemological beliefs have better critical thinking skills than students with lower level epistemological beliefs (King & Kitchener, 2004; Kuhn & Weinstock, 2002; Kuhn, 1999; 2005). Recently, Holma and Hyytinen (2014) have argued that there are several conceptual problems in this kind of hierarchical theory of knowledge (see also Elby & Hammer, 2001). In this section we focus on three conceptions of knowledge identified in the review of the literature on epistemology. These conceptions, specifically relativism, metaphysical realism and fallibilism, have theoretical importance for conceptualising critical thinking.

A relativist position implies that all knowledge is relative to the person who believes or that all interpretations, theories and beliefs are equally right. Because all beliefs are equally right, there is no reason to compare and evaluate different beliefs—all beliefs are equally justified (Holma, 2012; Holma & Hyytinen, 2014). The problem of relativism becomes clear when it is related to the concept of critical thinking (Holma & Hyytinen, 2014). Given that relativism allows people to construct their own ‘personal truths’, critical thinking turns out to be unnecessary (Bleazby, 2011). For example, there is no need to evaluate ideas or search for alternatives, because all ideas are equally trustworthy and justifiable (Bleazby, 2011; Holma & Hyytinen, 2014). Therefore, the idea that critical thinking presupposes the relativist view of knowledge is untenable.

Metaphysical realism is an epistemological position that assumes that ‘our knowledge and symbol systems [i.e. theories] directly reflect the structure of reality’ (Holma, 2004, p. 421; Putnam, 1981). The literature of personal epistemology seems to understand realism as metaphysical realism (see e.g. Kuhn 2005; Kuhn & Weinstock, 2002; see also Holma & Hyytinen 2014), and furthermore, it appears to connect with metaphysical realism the assumption of the possibility of the certainty of human knowledge. As King and Kitchener (2004) put it, knowledge is ‘obtained with certainty by direct observation’ (p. 7). [1] In the context of metaphysical realism, critical thinking turns out to be pointless.

Fallibilism is an epistemological position that implies that all our beliefs are liable to error (Reed, 2002; Niiniluoto, 1999; Holma, 2012). Contrary to relativism, fallibilism does not assume that all beliefs or theories are equally right. It presumes the possibility of improving our current conceptions, theories or beliefs. As Holma (2012, p. 399) aptly states of fallibilism, ‘this position, like the belief that all human knowledge is uncertain, coheres with the evolutionary understanding of knowledge: the bodies of knowledge we now have may be mistaken and thus [are] possible subjects for revision, but they have, nevertheless, survived the process of evolution to this point; as such, they provide the best available starting point for choices and action of the present moment concerning further inquiry’ (see also Peirce, 1934). From this point of view, epistemological fallibilism fits the presumption of critical thinking. Previous research on personal epistemology lacks the notion of epistemological fallibilism.

 Table 1.

Summary of the key concepts of this study

¹Based on Bailin & Siegel (2003); Ennis (1991); Fisher (2011); Fisher & Scriven (1997), Siegel (1988).

²Based on Scheffler (1965); cf. also Ryle (1949).
³Based on Pintrich (2002).

⁴Based on Holma (2004); Putnam (1981).

⁵Based on Holma (2012); Holma & Hyytinen (2014); Peirce (1934). 

Table 1 provides a summary of the definitions of the key concepts in this study. With this broader framework we are able to pin down different areas in critical thinking and epistemological beliefs, which have been shown to be vital for conceptualising these phenomena in prior studies or theorizations. Although the conventions of critical thinking and epistemological beliefs are commonly embodied in social practices (e.g. Arum & Roksa, 2011b; Elby & Hammer, 2001; Kuhn, 2005), the underlying dimensions (i.e. evaluating the reliability and relevance of evidence, identifying arguments, analysing information, addressing opposing viewpoints, reasoning) are relevant in each scientific discipline. Moreover, in line with previous studies we expected that students’ epistemological beliefs and critical thinking might vary within the same discipline (see Greene & Yu, 2014; see also Bailin & Siegel, 2003).

In our study we focused on the qualitative differences in critical thinking and personal epistemological beliefs by examining ten third-year university students’ thinking and performance in a cognitively-demanding authentic problem-solving situation. The aims of this study are twofold: to identify and describe qualitative differences in third-year university students’ critical thinking skills and epistemological beliefs in a problem-solving situation, and to analyse the interconnections between students’ personal epistemologies and critical thinking skills. To achieve these aims, we formulated the following research questions: (1) How are critical thinking and epistemological beliefs presented in a problem-solving situation in a specific group of third-year university students? (2) How do critical thinking and epistemological beliefs vary from one individual to the next?

6.         Research methods and materials

6.1       Participants

This study was conducted with ten third-year bioscience students drawn from the fields of biological and environmental sciences in a research-intensive university in Finland. The target population consisted of all third-year bioscience students in this particular university. First, we selected 40 students at random (approximately one-half of the target population). Then we invited all students selected to participate in our study. Ten out of 40 students volunteered. Seven of the participants were female and three male. The students’ ages varied from 22 to 29, the mean age being 24. All came from a homogeneous cultural background, and all shared the same first language (Finnish). In addition, the students had the same national high school certificate and had enrolled in the same bachelor’s study programme. The participants were at the same phase of their studies, that is, near the end of their bachelor’s studies, with the exception of one student whose study pace had been slower. During their university careers, the students had participated in lectures, practical laboratories, seminars, field courses and web-based teaching. We are aware that the sample size is too small for generalization. However, the purpose of this study is to deepen understanding of critical thinking and epistemological beliefs, for example, so as to describe how these phenomena vary across individuals in this specific group of students.

6.2       Procedures

For this study we collected a large body of data for each participant using a multi-method approach (Johnson, Onwuegbuzie & Turner, 2007), including think-aloud protocol, interviews and a Collegiate Learning Assessment (CLA) performance task. The data collection was carried out in the spring of 2010 and consisted of ten cognitive labs. The students came to a classroom and were given the details of the study. The students spent two to three hours reading and responding to the performance task. In responding to the task, the students were asked to verbalise their thoughts (to ‘think aloud’). In the course of carrying out the task while thinking aloud, the students were also asked to write a memorandum addressing critical issues in the task and recommending —and justifying— a course of action. Following the task, the students were interviewed about their processes in carrying out the task. Students were also asked questions about critical thinking, knowledge and knowing. Details of the procedures are provided below in appropriate sections.

6.2.1    Collegiate Learning Assessment (CLA)

The Collegiate Learning Assessment (CLA) instrument for assessing college-level critical thinking skills used in this study was developed by the Council for Aid to Education (CAE). The CLA is a standardised, open-ended test and it measures analytical reasoning, problem solving and written communication. Unlike most standardised tests used in measuring critical thinking, the version of the CLA used here did not include any multiple-choice questions (Klein, Benjamin, Shavelson & Bolus, 2007). The CLA consists of two elements: a set of performance tasks and a set of analytical writing tasks (Shavelson, 2010). Only the performance task was used in this study. Recent studies have found that open-ended problems with no obvious solution provide an opportunity for students to reflect on their beliefs about knowledge (Barzilai & Zohar, 2012; Ferguson & Bråten, 2012). For example, in a problem-solving situation students would need to determine the trustworthiness, and relevance, of different types of information presented to them, co-ordinate various pieces of information related to the problem and consider the underlying assumptions and claims (Shavelson, 2010).

The CLA performance task presents a realistic situation or problem and includes directions, open-ended questions and a document library containing reading material. In order to respond to the task, the students need to read, organise, synthesise and analyse information (which might be reliable/unreliable; relevant/irrelevant to the completion of the task; see Shavelson, 2010) from multiple documents (for example letters, memos, summaries of research reports, articles, diagrams, graphs, maps, interview notes). In doing these activities the students need to assess their confidence in information taken from various sources, including the relevance of the source, and thereby deal with conflicting information. They then need to decide on a course of action and provide a reasoned explanation and justification for their course, drawing on supporting information from the document library (Klein et al., 2007; Shavelson, 2010). They also have to argue for and against alternative explanations. The specific performance task used in this study is proprietary and consequently cannot be described here. An example of a representative CLA performance is presented in Figure 1.

Figure 1. (see pdf) An example of a CLA performance task. Adapted from R. Shavelson, 2010, Measuring College Learning Responsibly: Accountability in a New Era. Stanford, CA: Stanford University Press, p. 38. 

6.2.2    Cognitive labs

The purpose of cognitive labs is to study the cognitive processes that students use when they complete different tasks. Students are asked to report their thoughts verbally as they carry out a task (see Johnstone, Bottsford-Miller & Thompson, 2006). In this study cognitive labs were divided into three parts: (1) instruction and training, where the researcher explained what the cognitive lab was about and trained the students to think aloud with a short warm-up task; (2) ‘think-aloud’, where the students talked aloud while completing the CLA performance task; and (3) a follow-up interview. The cognitive lab for each student was video-recorded and lasted two to three hours. To ensure the consistency of cognitive labs, a script of directions and the same training task and the interview questions for each student were used. The videos were recorded with two cameras and a table microphone. The cognitive workshop produced the following materials: video data, content logs (see below), written test answers and transcribed interview data.

The neutral type of think-aloud protocol conducted by Ericsson and Simon (1993) in which students were not interrupted while they were performing a task was used in this study. The think-aloud method makes it possible to collect data about a student’s ongoing thinking processes whilst he or she is working on a task  (Ericsson & Simon, 1993; Cotton & Gresty, 2006; van Someren, Barnard & Sandberg, 1994). We assume that students’ ‘knowing-that’ attributions (e.g. ‘scientific knowledge is true’) may reflect their epistemological orientations and reveal their criteria for justifying beliefs (see Scheffler, 1965). Moreover, in some cases the think-aloud method makes it possible to explore critical thinking in action, especially in situations that simulate real-world circumstances.

Immediately after the task was performed, a follow-up interview was conducted. The aim of the interview was to gain more detailed information about the processes and knowledge that the students used to complete the task and to probe students’ beliefs about knowledge and knowing. For example, the students were asked questions about how they dealt with conflicting information, how they decided which information to use, what sources of information in documents from the documents library they trusted and why, and how they usually evaluate knowledge.

7.         Data analysis

The data were analysed using a qualitative thematic analysis with an abductive approach (Timmermans & Tavory, 2012; Haig 2005). An abductive strategy means that the themes identified from the data were linked to the theoretical understanding based on previous studies. Abduction is a process that combines things which one had not previously associated by creating a new interpretation, that is, the relationship of a new combination of study features (Timmermans & Tavory, 2012). Hence, the analysis process was nonlinear, moving back and forward amongst all the data, data items, analysed qualities and understanding of the phenomenon based on prior studies. The first and fifth authors were responsible for the analysis, but the final results were obtained through a thorough discussion with all authors. The data were processed in such a way that the participants could not be identified.

The analysis included four phases (Figure 2) that represented the unique combination of data-grounded and theory-driven phases, as well as phenomenon and individual-level analyses. During the first phase, video recordings were initially indexed with the ELAN program, which allows the addition of as many tiers and annotations on the video stream as needed (see Lausberg & Sloetjes, 2009; Max Planck Institute for Psycholinguistics, 2012). The purpose of indexing was to make the large video data set easier to handle. In this study the indexing tiers corresponded to the parts of cognitive labs including training, think-aloud methods and interviews. In addition, students’ interviews from the videos were transcribed.

*Based on Braun & Clarke (2006, p. 87).

Figure 2. (see pdf)  A visualisation of the analysis process.

After the indexing, content logs were created for each video in which accurate descriptions and summaries of events were systematically recorded. Transcriptions of relevant sections of verbalisations of students’ critical thinking and epistemological beliefs (e.g. whenever a student evaluated the quality and reliability of the information in a document or where a student reached a conclusion based on her or his analysis) and nonverbal acts (e.g. a student did not read in detail or skipped over the document) were also included in the log (cf. Table 1).

The second phase of the analysis was the data coding (see Table 2 for definitions). This phase was theory-driven, meaning that the features guiding the coding were based on prior studies (see Table 1). The coding focused on the following qualities: the process by which the student approached the task and solved the problem, the knowledge that the student used to carry out the task, the critical thinking exhibited, and epistemological beliefs. These different qualities were coded systematically across the entire data set and within the data items such as the transcribed interviews and the think-aloud videos of each person. By this means, all the data items from one student, including the video data, content log, written test answers and transcribed interviews, were coded and analysed separately, after which data from all students were combined and compared (see Table 3 for an example of the codes). All extracts were labelled with a student code (S1-S10) and a method code (I= interview, T=think aloud, W= written test answer). The data examples were translated into English.

Table 2.

Data sources and focal points of coding

Table 3.
An example of codes

In the third phase the codes and coded extracts were grouped under potential themes, and all the relevant data were gathered under each theme (Braun & Clarke, 2006). We identified a variety of preliminary themes on the basis of the codes. During the analysis, the preliminary themes were defined and combined several times. In the end two main themes and two subthemes remained (see Figure 2). The final themes were refined, labelled and cross-checked to see if they worked in relation to the coded extracts and the entire data set. The focus of the thematic analysis was the variation of study features on the phenomenon level.

After completing the thematic analysis, we found that the students could be placed in different profiles based on our themes as well as on patterns of behaviour and cognition observed. This phase focused on the variation of study features at the individual level. Thereafter, we conducted final descriptions, interpretations and revisions of the results. The results of thematic analysis show how critical thinking and epistemological beliefs manifested themselves in this particular group of students, whereas the student profiles describe how these phenomena vary across individuals.

8.         Results

In the thematic analysis two main themes were identified: (1) flexibility in critical thinking and (2) variation in critical thinking and epistemological beliefs. The two themes emerged from exploring the students’ critical thinking from different perspectives. The ways in which the themes were related differed amongst the participants, which further allowed us to identify student profiles. We identified two main profiles, and on the basis of their characteristic features we labelled them as (1) thorough processing and (2) superficial processing. The results are described using a combination of identified themes and student profiles.

8.1       Flexibility in critical thinking

Students showed various skills in their ability to adapt their thinking and their performance flexibility to the demands of the task. There was clear variation in the students’ ability to change their actions or ways of critical thinking, in which we identified both rigidity and flexibility. Flexibility meant that the students could modify their actions and processes and change their behaviours as needed, whereas rigidity refers to situations in which students could not change their processes or look at things from a new perspective or adjust to new evidence in a problem-solving situation. Students who were able to make changes in their actions showed open-mindedness and an inquiring attitude.

In the following extract, one student describes how he adjusted his performance and ended up analysing and interpreting the documents correctly:

I approached [this assignment] maybe a little too much as if I had simply copied what they say here in these papers and put them down [in my answer]. But then when I started thinking, like about my own views on the topics, then right off in [question] number one, it took me a really long time to answer this question. (S8I)

On the other hand, there were students who could not adjust their thinking or performance. Some of these students said that they always act in the same way:

Well, I’m always like this time-management catastrophe. Like in exams and everything, especially exams, it always feels like I run out of time. And  in general I notice that in all comprehension and analysis assignments and things like that, they always take me a really long time. (S5I)

8.2       Variation in critical thinking and epistemological beliefs

Students showed various aims in the problem-solving situation. Some students tried to understand the complex situation, whereas others tried to find the right answer to the problem. Students also varied in their critical thinking, including (a) their disposition and ability to identify, analyse, evaluate and interpret information; (b) their ideas and arguments in judging the acceptability of documents; (c) their abilities to recognise relationships between assumptions; (d) their abilities to make a reasoned decision; and (e) their abilities to produce explanations and arguments. In addition, students’ epistemological beliefs varied. Some students claimed that only through scientific knowledge we can arrive at truth. However, other students expressed the idea that both objective and subjective knowledge can hold the highest epistemic status.

We found that critical thinking emerged as a tool for understanding knowledge and determining the goodness and reliability of knowledge; thus, students’ epistemological beliefs were interwoven into their critical thinking. Within this theme we found that students used critical thinking either a) as a tool for enhancing understanding or b) as a tool for determining truth or falsehood. Based on this difference, students could be classified in one of two qualitative student profiles, either (1) thorough processing or (2) superficial processing. The profiles captured the diversity of the students’ abilities and dispositions to think critically. In addition, these two profiles characterised the variation in how students viewed the nature and limitations of knowledge and knowing, and especially in how they determined what is needed to evaluate knowledge as true or justified and how they acquired and used the knowledge in the problem-solving situation (see Table 4). The phrase ‘acquiring knowledge’ here emphasises the dominant way that students used to obtain knowledge in a problem-solving situation.

The students classified in the profile called ‘thorough processing’ demonstrated an ability to carry out a deep processing of the content of the documents. These students saw knowledge as fallible and contextual. Similarly, the students in the profile called ‘superficial processing’ expressed the idea that knowledge is fallible, yet they did not consider the contextual nature of knowledge at all. In the problem-solving situation they did make a serious effort to analyse, interpret or synthesise the information in the materials. The thorough processing profile is further divided into two sub-profiles: (1A) reasoning in order to reach conclusions and (1B) intuition. Likewise, the second profile, ‘Superficial processing’, also consisted of two sub-profiles: (2A) referring to an argument made by authoritative specialists or experts and (2B) trust in scientific method and proof. We describe the characteristics of the profiles and sub-profiles below and provide details pertaining to variation in academic thinking.

Table 4.

The nature of knowledge and acquiring knowledge in two qualitatively different student profiles of critical thinking

8.3       Profile 1: Thorough processing students

The students (n=5) who deeply analysed the content of the documents created their own understanding of the problem-solving situation. For them, critical-thinking skills were tools to deepen and enhance understanding. These students believed that theories and beliefs could be understood in relation to some context, as the following extract shows:

- - yeah, I don’t believe [the chair of the stakeholder group] is completely off the mark either. [The reliability of] knowledge is just always context-specific. (S8I)

These students considered it possible to improve current theories and beliefs. These students were thus open to new evidence that could disprove a previously-held position or belief. For them, scientific knowledge is probably reliable. They believed that both objective and subjective knowledge could attain the highest epistemic status, meaning that subjective perceptions (e.g. their own perceptions or information obtained from someone else) could also be reliable. These students thought that the credibility of knowledge could be affected by vested interests or bias, for example. Although these students emphasised their own role in constructing knowledge, they did not believe that all knowledge is constructed or generated by human minds. From the epistemological perspective, these students took the fallibilist position.

The students who belonged to the ‘thorough processing’ profile were further divided into two sub-profiles on the basis of how they acquired knowledge and reached conclusions in the problem-solving situation and how flexible they were in changing their actions or ways of thinking. The first sub-profile was called (1A) reasoning in order to reach conclusions and the second was called (1B) intuition.  

8.3.1    Reasoning in order to reach conclusions

Two students endeavoured to reach conclusions by reasoning. These students analysed connections across the information presented in the different documents. They also clarified and interpreted different claims and ideas that were presented in the documents. On the basis of their own analyses, they synthesised information, reached a clear decision or conclusion, provided arguments for their decision and explained why this decision was the best in light of all the issues brought up in the documents. In the following example one student describes her analytical process:  ‘Somehow I knew how to read beyond [the documents]’ (S4I). These students were also able to adjust their thinking in line with new evidence and make changes in their actions. These students justified conclusions with good reasons (e.g. reliable and valid evidence) and considered themselves as active and responsible knowers, as the following extracts show: 

But maybe I wouldn’t, like, start criticising right away; somehow, I’d have to start looking into, you know, on what basis they arrived at these figures. (S10I)

For instance, using this graph is fine, but I think it’s been, you know, clearly misinterpreted here in the text. (S4I)

These students created their own understandings of the situation on the basis of their analyses. They used the materials for the analysis and evaluation process in a way that went beyond the obvious. For example, they identified, analysed, evaluated and interpreted all the major facts and ideas presented in the documents. They consciously excluded some information in the documents because of contradictory evidence. In addition, they were able to distinguish relevant claims from irrelevant ones. These students also judged the reliability of the documents, evaluated presuppositions and analysed connections between claims. Furthermore, they produced different explanations, identified reasons, produced arguments and drew inferences.

These students further identified and used several criteria in evaluating reliability: corroborating claims from different sources, evaluating the context in which the claim was made, exploring who interpreted the data and evaluating the presuppositions. Moreover, these students considered the ethical aspects of knowledge: knowledge and information shape human beings’ worldview.

I’ve just gotten the impression about newspapers, about the media too, that it somehow has the effect that the opinions [presented in them] are so strong that maybe you don’t analyse it so clearly. So like even Helsingin Sanomat [Finland’s largest daily] really has, somehow it seems that they have a pretty strong, you know, bias... you know that even if it’s neutral in a way, then the fact the issues they raise in it, in a way that already affects what information is raised, and what... that it like really powerfully shapes people’s worldview. (S4I)

8.3.2    Intuition

Three students justified conclusions by intuition. These students created their own understanding of a situation. However, they did not select materials or question any information: they used all the information in the documents, such as empirical knowledge, expert opinions, reports, maps, experiences of an inhabitant, recommendations, letters and second-hand knowledge. These students acquired knowledge in a rather uncritical way. They rarely evaluated the reliability of documents. Indeed, these students did not have clear criteria for evaluating the reliability or relevance of information. They just trusted their intuition:

This just seems scientific somehow. (S6T)

I don’t know how I should formulate this, but I’ll start by saying that when I read, for instance... or when I’m taking classes, I don’t spend a whole lot of time wondering if some piece of information is reliable or not. (S6I)

These students started to analyse and interpret thoroughly all information presented in the documents. They identified all major facts and ideas. They also considered different decisions or explanations, but could not explain what decision was the best or why. There were too many options available. Because the students did not reach clear conclusions, they did not present any arguments for accepting the conclusion either. These students showed an inability to adjust their thinking to new evidence or make changes in their actions.

8.4       Profile 2: Superficial processing students

Common to all students in the second main profile was that they processed the materials in the problem-solving task superficially: they did not make a serious effort to analyse, interpret or synthesise the information in the materials. This profile consisted of five students who used critical thinking as an instrument for determining truth or falsehood. Their goal was to find the right answer to the problem. In contrast to the ‘thorough processing’ profile, students in this profile believed that knowledge is trustworthy only if it was produced through a reliable process, for example, by using empirical methods or consulting suitable experts. For these students, scientific and verified knowledge is the most reliable, because that kind of knowledge is based on evidence, and it is unbiased and objective. The students believed that subjective knowledge is predominantly untrustworthy. However, these students considered empirical knowledge (which holds the highest epistemic status) to be fallible too, not absolutely certain. They believed that some theories might be false and that it is possible to improve current conceptions and theories. From the epistemological perspective, these students also took the fallibilist position.

The analysis indicated varying problems in critical thinking, such as problems in evaluating information, reasoning and reaching conclusions. Some of these students also had little motivation to think critically. Characteristic of the students in this profile was that they focused on isolated details. They took knowledge for granted. In other words, they accepted knowledge (particularly scientific knowledge) as true without question. These students were further divided into two sub-profiles according to how they acquired knowledge in a problem-solving situation, trusting either (2A) an argument by authoritative specialists or experts and (2B) verified empirical evidence or testimony.

8.4.1    Referring to an argument by authoritative specialists or experts

Two students were categorised in this sub-profile. These students trusted authorities in acquiring knowledge. They saw themselves as uncertain knowers. These students believed that if a person who is said to be an authority on something makes an argument about that something, then the argument should be trustworthy and therefore usable. The right answers can be reached by consulting the right expert. These students repeated arguments and conclusions as these were presented in the documents. They drew on empirical knowledge and expert opinions, that is, arguments from authoritative sources.

These students had difficulties in evaluating information. They focused on details and took in all the information they were presented without question. They picked up isolated and obvious details from the materials for each question. The students did not properly analyse, evaluate or interpret the information presented in the documents; they just jumped to conclusions. They disregarded and seriously misinterpreted important information. They also had problems in reasoning and reaching a conclusion.  In order to make decisions or arguments, these students reproduced lists of isolated details from documents. They did not provide any reasons or explanations for their decisions. Moreover, they did not identify alternative solutions. These students presented some unreliable claims as being credible. In the interview one student representing this sub-profile said that she has had similar problems in learning:

Creative comprehension and, like, reaching a synthesis of overall concepts is really challenging for me. Like, for instance, it’s really hard to study for exams, because I’d be more than happy to read the book, but then I don’t really grasp the key message and structure that it’s trying to communicate. Acquiring data independently and, like, learning information that way is challenging. So, for instance, I haven’t done all my exams. I haven’t done them because, I’ve tried to start [studying for] them lots of times, but then some, how would you put  it, if listening is auditory, then learning from text is pretty hard for me. This third year, which is currently underway, has been, like, really hard. I’ve really haven’t gotten many credits. I don’t feel I’ve accumulated the amount of information I should have or could have in three years. That the pieces of information are discrete and still pretty scattered in my head at the moment. (S5I)

These students expressed the view that knowledge is always uncertain, but they did not consider themselves capable of evaluating knowledge. These students named a few external criteria for evaluating knowledge (such as an authority, expert opinion, publication, openness, journal citations). However, in practice they did not know how to use these criteria independently. Both of these students gave authoritative experts the responsibility for evaluating knowledge, as the following example demonstrates:

I don’t know what the right approach is in order to grasp those overall concepts from that huge mass of teensy-weensy details. Because a candidate has to read a huge number of articles to find the ones that are, like, related to one’s own topic and all. So it’s really hard when you’re, like, reading an article to judge why this one might be better than that one. So. But I got a tip from my supervisor that I should pay attention to the reliability of the journal. To be honest, it’s the research articles, the ones we have at the university, that are actually the only ones we’re told we can cite. And then it’s like... they’re easy to evaluate based on which publications are more credible. And on the Web on [sic] Science, they have this one like... what is it, like an indicator that they have, just based on the number of citations and other factors, of the accuracy of the research data… It’s hard! In a way, to make that distinction between what’s true and what isn’t. At least I don’t have the know-how to say what’s true and what isn’t. (S5I)

Both students expressed the view that in a real-life situation they would seek help from other people, such as authoritative specialists (e.g. a university teacher) or other students. For example, one student representing this sub-profile said several times that she needed co-operation with other students to solve the task:

I haven’t really had to do anything like this before. That it’s pretty hard in a way. There are so many points and, you know, perspectives here. I haven’t even had to think about stuff like this at the university, then it’s really like new for me, or you know. The assignment was pretty difficult. This might have been more interesting as a group assignment. Like  there would have been, you know, interaction, and then maybe it would have generated more thoughts somehow. (S7I)

8.4.2    Trust in scientific method and proof

The three students comprising this sub-profile were very critical. They all selected documents roughly based on empirical evidence, excluding more than half of the documents provided. These students were aware of their own behaviour:

I eliminate some of the documents right away, for instance, email exchanges and letters, because the people haven’t investigated the matter; the text was written based on a gut feeling. (S3T)

These students expressed the view that scientifically and empirically verified knowledge is the most reliable. They knew that corroboration from other reliable and related sources improves credibility. In the problem-solving situation they only trusted and used arguments by scientific authorities. These students described themselves as ‘error seekers’ in the interviews. The following examples illustrate the view of the students in this sub-group:

I trust exam books and articles a lot, yes. The difference between the two is that books can often, you know, be unreliable. Plus the fact that, at least when they’re academic, it has a lot to do with when they were written, because things move so fast. That I have this one book for my thesis that I was just looking at, it’s got tons of mistakes. So, like, you just have to find them yourself. But with articles, probably those, and then of course depending on the journal. That maybe some article in Science: I consider them pretty reliable. Nowadays, I’m a little too sceptical about all kinds of things. I question a lot more these days than I used to. (S1I)

You can get the first impression [of reliability], of course, from the kind of source it was published in. In other words, I wouldn’t swallow some Iltalehti [a Finnish tabloid] headline on some scientific subject without thinking it over properly first. But having a reference to those academic publications, and as far as how I’ve drawn those conclusions myself after having read the article, not based on some newspaper headline, then that would be at least important in terms of first impressions. And… well, even if you read a scientific article, if it doesn’t agree at all with what you’ve learned about the topic earlier, then of course you’d have good cause to suspect those research results quite a bit. But the source is what I’d probably consider as the main thing. (S3I)

Although these students describe themselves as critical, they did not evaluate information from reliable sources in the problem-solving situation. For example, they did not recognise that two sources, which included empirical or verified knowledge, were biased. They analysed and interpreted information superficially and focused on isolated details. They did not interpret the documents they selected nor did they consider presuppositions. In order to draw conclusions these students mainly reproduced details from the documents. They did not identify alternative solutions or conclusions or approaches to the problem. Nor did they provide any reasons or explanations for their own conclusions. They identified only a few claims that were presented in the documents and disregarded many relevant aspects of those claims. As a result they had problems reaching a conclusion. In the following example one student described the situation as follows: ‘At least I wouldn’t draw any conclusions based on those [documents]’ (S1T). All these students thought that there was one definite answer to the problem.

One student in this sub-profile emphasised that she does not have any disposition or motivation to express reasons for or against some idea in a test situation or in everyday life:

In everyday life it’s rare that, if you’re discussing something, it’s rare that anything like this happens. Or I never, really rarely discuss anything argumentatively in any way. In real life I simply don’t like it, discussing issues. (S9I)

8.5       Summary of the results

Figure 3 combines the two main themes in order to form a comprehensive picture of participants’ critical thinking. Students who had several problems in critical thinking, yet had flexibility coped with the demands of the task. For example, two students had problems evaluating documents and did not form a general picture of the situation presented in the documents. Because the students were struggling with the demands of the task, they selected documents and reproduced arguments and conclusions just as these were presented in the materials. Eventually, the students reached a limited conclusion. On the other hand, there were students who were skilled in specific critical-thinking skills, such as analysing and interpreting information, but lacked other abilities, such as evaluating conflicting claims or producing explanations. These students could neither reach a conclusion nor were they able to determine the weaknesses of alternative solutions. In addition, these students were unable to change their actions or thinking; for example, they were not flexible in time management. These students somehow ‘over-analysed’ the problem, and, in the end, they failed in the problem-solving process.

In sum, the aspect that distinguished the participants were the differences in 1) aims, 2) the skills and disposition to think critically, 3) epistemological beliefs, 4) acquiring knowledge and 5) the skill of flexibility in adapting thinking and performance to the demands of the task.

Figure 3. Summary of results.

Even though the number of participants in this study was small, the variety in the students’ critical thinking was evident. Our results showed that after three years of university study, students’ critical-thinking skills and epistemological beliefs differed greatly, and eight out of ten volunteer students had some problems in critical thinking (cf. Arum & Roksa, 2011; Pascarella, Blaich, Martin & Hanson, 2011). The multi-method approach effectively revealed the variety of problems that university students may encounter. While many problems were related to the lack of disposition or skill, such as an inability to evaluate the credibility of documents, examine presuppositions, make interpretations, develop a personal perspective or generate arguments or conclusions, some of the problems were related to an inability to modify the whole critical-thinking process in a flexible manner. These findings corroborate the ideas of Fisher (2011) and Scheffler (1965), who suggested that individuals may be more or less skilled at different critical thinking abilities. In other words, a student may have the ability to identify and evaluate information, for example, yet at the same time struggle with other abilities, such as arriving at a conclusion, adjudicating conflicting claims or producing arguments. Therefore, it is clear that unilateral instructions concerning critical thinking are difficult to provide.

The findings of this study support the idea that students’ epistemological beliefs were interwoven into their critical thinking (cf. Kuhn, 2005; Kuhn & Weinstock, 2002). Critical thinking emerged as a tool for understanding and determining the relevance and reliability of knowledge. Students who showed superficial processing believed that objective knowledge (i.e. scientific and verified knowledge) has the highest possible epistemic status. Although it is sensible to trust in scientific and empirical knowledge more than in personal opinions, the problem was that these students accepted scientific knowledge without question: they did not analyse, evaluate or interpret the information contained in the documents they were given. They acquired knowledge by appealing in equal measure to authoritative opinion, trusting in verified empirical evidence and listening to testimonies. These students palmed off a justification for knowing on authoritative experts.

Contrary to the results of many previous studies (e.g. Kaartinen-Koutaniemi & Lindblom-Ylänne, 2012; King & Kitchener, 2004; Kuhn, 1999, 2005; Kuhn & Weinstock, 2002), our main finding was that the students who appealed to authorities, testimonies or empirical evidence did not believe that knowledge is absolutely certain or unquestionable. Nor did these students share the view that beliefs accurately represent or correspond to reality. In effect, the students did not share a sense of metaphysical realism. Instead, these students claimed that scientific theories are uncertain, but probably true. The findings also show that the students who believed that knowledge is contextual and relative did not share a relativist view of knowledge. This finding is also contrary to the findings of earlier studies (e.g. Lahtinen & Pehkonen, 2012). Conversely, all of the students saw knowledge as fallible. The students believed that it is possible to seek criteria for evaluating, comparing and justifying beliefs or theories. Although some students struggled with evaluating knowledge, all of them saw current conceptions and theories as a starting point for further inquiry. They were thus fallibilist in the epistemological sense.

This study further shows that students’ belief in themselves as critical thinkers and knowers is not necessarily equivalent to how they perform. Thus, we assume, along with previous studies (Elby & Hammer, 2001; Greene & Yu, 2014), that the self-reported assessment method is not enough to gauge these kinds of complex processes. The present small-scale qualitative study has provided a unique picture of the critical thinking and personal epistemological beliefs of ten third-year bioscience students. Furthermore, this study has educational significance by revealing problems in these students’ critical-thinking skills and by describing the role of students’ conception of knowledge in the process of thinking critically. Through a multifaceted approach, it was also possible to deepen understanding of the emphases and gaps in the prevailing empirical research on critical thinking and personal epistemology. However, the findings of this study should not be interpreted as an accurate prediction of the target population. The findings of this study rather illustrate the nature of the phenomenon being studied, and how the different aspects of critical thinking and epistemological beliefs are intertwined and contribute to it together. This study involved a small, homogeneous sample of students in one discipline only. Owing to these limitations, more communication between the theoretical, empirical and methodological perspectives is required to increase understanding of this complex phenomenon in the different spheres.

Keypoints

•               In this exploratory study we provide a multidimensional framework for analysing critical thinking by combining theoretical aspects from philosophical, educational and psychological approaches.

•               In this exploratory study we provide a multidimensional framework for analysing critical thinking by combining theoretical aspects from philosophical, educational and psychological approaches.

•               A large body of data for each participant (n=10) was collected using multiple methods, including think-aloud protocol, interviews and a Collegiate Learning Assessment (CLA) performance task.

•               The result shows that students’ epistemological beliefs were interwoven into their critical thinking: students used critical thinking as a tool for enhancing understanding and for seeking a right answer

•               None of students shared an absolutist view of knowledge.

•               None of students shared a relativist view of knowledge.

•               All students shared a fallibilist view of knowledge.

Acknowledgements

The first author was financially supported by a scholarship from the Alfred Kordelin Foundation. The authors are grateful to their colleagues for helpful comments on an earlier version of this article in manuscript, as well as to the students for their participation. We are grateful to Roger Benjamin at the Council for Aid to Education for permitting us to use a performance task from the Collegiate Learning Assessment and to Viivi Virtanen (PhD) and Mikael Kivelä (MA) for helping us in data collection.

References

The American Philosophical Association. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction "the Delphi Report". Committee on Pre-College Philosophy. Millbrae, CA: The California Academic Press.

Angeli, C., & Valanides, N. (2009). Instructional effects on critical thinking: performance on ill-defined issues. Learning and Instruction, 19, 322-334.

Arum, R., & Roksa, J. (2011a). Limited learning on college campuses. Society, 48, 203-207. DOI 10.007/s12115-011-9417-8

Arum, R., & Roksa, J. (2011b). Academically adrift. Limited Learning on college campuses. Chicago: the University of Chicago Press.

The Australian Council for Educational Research. (2001). Graduate Skills Assessment. Summary Report. 01/E Occasional Paper Series. Higher Education Division. Retrieved from http://www.acer.edu.au/documents/GSA_SummaryReport.pdf

Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2007). Evaluating assessment quality in a competence-based education: A qualitative comparison of two frameworks. Educational Research Review 2, 114-129.

Bailin, S., & Siegel, H. (2003). Critical thinking. In N. Blake, P. Smeyers, R. Smith, & P. Standish (Eds.), The Blackwell guide to the philosophy of education (pp. 181-193). Blackwell Publishing.

Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: evaluating and integrating online sources. Cognition and Instruction 30, 39-85.

Bleazby, J. (2011). Overcoming relativism and absolutism: Dewey’s ideals of truth and meaning in philosophy for children. Educational Philosophy and Theory 43, 453-466. doi: 10.1111/j.1469-5812.2009.00567.x

Bok, D. (2006). Our underachieving colleges. A candid look at how much students learn and why they should be learning more. Princeton, NJ: Princeton University Press.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology 3, 77-101.

Cotton, D., & Gresty, K. (2006). Reflecting on the think-aloud method for evaluating e-learning. British Journal of Educational Technology 37, 45-54.

Dewey, J. (1910). How we think. Boston: D.C. Heath & Co.

Dierick, S., & Dochy, F. (2001). New lines in edumetrics: new form of assessment lead to new assessment criteria. Studies in Educational Evaluation 27, 307-329.

Elby, A., & Hammer, D. (2001). On the substance of a sophisticated epistemology. Science Education, 554-567.

Ennis, R. (1991). Critical thinking: a streamlined conception. Teaching Philosophy 14, 5-24.

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, Ma: MIT Press.

Everitt, N., & Fisher, A. (1995). Modern epistemology: a new introduction. New York, McGraw-Hill.

Fenstermacher, G. D. (1994). The Knower and the Known: The Nature of Knowledge in Research on Teaching. Review of Research in Education 20, 3-56.

Ferguson, L. E., & Bråten, I. (2012). Students’ profiles of knowledge and epistemic beliefs: Changes and relations to multiple-text comprehension. Learning and Instruction 25, 49-61.

Fisher, A., & Scriven, M. (1997). Critical thinking: its definition and assessment. Edgepress and Centre for Research in Critical Thinking. University of East Anglia.

Fisher, A. (2011). Critical thinking: an introduction. Second edition. Cambridge: Cambridge University Press.

Green, J. A., & Yu, S. B. (2014). Modelling and measuring epistemic cognition: A qualitative re-investigation. Contemporary Educational Psychology 39, 12-28.

Haig, B. D. (2005). Abductive theory of scientific method. Psychological Methods 10, 371-388.

Halpern, D. F. (2014) Thought and Knowledge. Fifth edition. NY: Psychology Press.

Hammer, D., & Elby, A. (2003). Tapping epistemological resources for learning physics. The Journal of the Learning Sciences 12 (1), 53-90.

Heijltjes, A., van Gog, T., Leppink, J., & Paas, F. (2014). Improving critical thinking: Effects of dispositions and instructions on economics students’ reasoning skills. Learning and Instruction 29, 31-42.

Hofer, B. K. & Pintrich, P. R. (2002). Personal epistemology: The psychology of beliefs about knowledge and knowing. New Jersey: Lawrence Erlbaum Associates.

Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist 39, 43-55.

Hofer, B. K. (2005). The Legacy and the challenges: Paul Pintrich’s contributions to personal epistemology research. Educational Psychologist 40, 95-105.

Hofer, B. K. (2006). Domain specificity of personal epistemology: resolved questions, persistent issues, new models. International Journal of Educational Research 45, 85-95.

Holma, K. (2004). Plurealism and education: Israel Scheffler’s synthesis and its presumable educational implications.  Educational Theory 54, 419-430.

Holma, K. (2012). Fallibilist pluralism and education for shared citizenship. Educational theory 62, 397-409.

Holma, K. (2014). The Critical Spirit: Emotional and moral dimensions critical thinking. Under review.

Holma, K., & Hyytinen, H. (equal contribution) (2014). The philosophy of personal epistemology. Under review.

Ichikawa, J. J., & Steup, M. (2012). The Analysis of Knowledge. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy (Winter 2012 Edition). Retrieved from http://plato.stanford.edu/archives/win2012/entries/knowledge-analysis/>.

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research 1, 112-133.

Johnstone, C. J., Bottsford-Miller, N. A., & Thompson, S. J. (2006) Using the think aloud method (cognitive labs) to evaluate test design for students with disabilities and English language learners. Technical Report 44. Minneapolis, MN: University of Minnesota, National Centre on educational Outcomes. Retrieved from http://www.cehd.umn.edu/nceo/OnlinePubs/Tech44/

Kaartinen-Koutaniemi, M., & Lindblom-Ylänne, S. (2008) Personal epistemology of psychology, theology and pharmacy students: a comparative study. Studies in Higher Education 33, 179-191.

Kaartinen-Koutaniemi, M., & Lindblom-Ylänne, S. (2012). Personal Epistemology of University Students: Individual Profiles. Education Research International 2012, 1-8. DOI:10.1155/2012/807645

King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical thinking in adolescents and adults. San Francisco: Jossey-Bass.

King, P. M., & Kitchener, K.S. (2004). Reflective Judgment: Theory and Research on the Development of Epistemic Assumptions Trough Adulthood. Educational Psychologist 39, 5-18.

Klein, S., Benjamin, R., Shavelson, R., & Bolus, R. (2007). The collegiate learning assessment, facts and fantasies. Evaluation Review 31, 415-439. Doi: 10.1177/0193841X07303318.

Klein, S., Freedman, D., Shavelson, R., & Bolus, R. (2008). Assessing School Effectiveness. Evaluation Review 32, 511-525.

Kuhn, D. (1999). A Developmental Model of Critical thinking. Educational Researcher, 28, 16-25.

Kuhn, D. (2005). Education for thinking. Harvard University Press.

Kuhn, D., & Weinstock M. (2002). What is epistemological thinking and why does it matter? In Barbara K. Hofer & Paul, R. Pintrich (Eds.) Personal epistemology: The psychology of beliefs about knowledge and knowing, 121-144. New Jersey: Lawrence Erlbaum Associates.

Lahtinen, A-M, & Pehkonen, L. (2013). ‘Seeing things in a new light’: conditions for changes in the epistemological beliefs of university students. Journal of Further and Higher Education, 37, 397-415.

Lausberg, H., & Sloetjes, H. (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, Instruments, & Computers, 41(3), 841-849. Retrieved from http://www.springerlink.com/content/d53722q3k3314374/

Maclellan, E. (2004). How convincing is alternative assessment for use in higher education. Assessment & Evaluation in Higher Education 29, 311-321.

Markowitsch, J., & Messerer, K. (2007). Practice-oriented methods in teaching and learning in higher education. In P. Tynjälä, J. Välimaa, & G. Boulton-Lewis (Eds.), Higher education and working life: collaborations, confrontations and challenges (pp. 177-194). Oxford: Elsevier.

Max Planck Institute for Psycholinguistics. (2012). ELAN. [Software]. Available from http://tla.mpi.nl/tools/tla-tools/elan/

Moore, T. (2004). The critical thinking debate: how general are general thinking skills. Higher Education Research & Development 23, 3-18.

Moore, T. (2013). Critical thinking: seven definitions in search of a concept. Studies in Higher Education, 38, 506-522.

Muis, K. R., Bendixen, L. D., & Haerle, F. C. (2006). Domain-Generality and Domain-Specificity in Personal. Epistemology Research: Philosophical and Empirical Reflections in the Development of a Theoretical Framework. Educational Psychology Review 18, 3–54.

Niiniluoto, I. (1999). Critical Scientific Realism. Oxford: Oxford University Press.

Pascarella, E. T., Blaich, C., Martin, G. L., & Hanson, J. M. (2011) How robust are the findings of academically adrift? Change: The Magazine of Higher Learning 43, 20–24.

Peirce, C. S. (1934). Pragmatism and Pragmaticism, vol. 5 of Collected Papers of Charles Sanders Peirce. Cambridge, Massachusetts: Harvard University Press.

Perry, W. G. (1970). Forms of Intellectual and Ethical Development in the College Years. Harvard University.

Pintrich, P. R. (2002). Future challenges and directions for theory and research on personal epistemology. In B. K. Hofer & P. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 121-144). New Jersey: Lawrence Erlbaum Associates.

Putnam, H. (1981). Reason, Truth and History. Cambridge: Cambridge Press.

Reed, B. (2002). How to think about fallibilism. Philosophical Studies 107, 143-157.

Ryle, G. (1949). The concept of mind. London: Hutchinson’s University Library.

Scheffler, I. (1965). Conditions of knowledge. An introduction to epistemology and education. Glenview: Scott, Foresman and Company.

Shavelson, R.J. (2010). Measuring college learning responsibly: Accountability in a new era. Stanford, CA: Stanford University Press.

Shope, R. K. (2004). The Analysis of Knowing. In I. Niiniluoto, M. Sintonen, & J. Woleński (Eds.) Handbook of Epistemology (pp. 3-56). Dordrecht: Kluwer Academic.

Siegel, H. (1988). Educating Reason: Rationality, Critical Thinking, and Education. NY: Routledge.

Smith, G. (2002). Are There Domain-Specific Thinking Skills? Journal of Philosophy of Education 36, 207-227.

van Someren, M.W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method. A practical guide to modelling cognitive processes. Department of Social Science Informatics. University of Amsterdam. London: Academic Press.

States, A., Min-Leliveld, M., Gijbels, D., & Petegem, P. van (2010). The impact of instructional development in higher education: The state-of-the-art of the research. Educational Research Review 5, 25–49.

Timmermans, S., & Tavory, I. (2012). Theory construction in qualitative research: from grounded theory to abductive analysis. Sociological Theory 30, 167-186.

Tremblay, K., Lalancette, D., Roseveare, D. (2012) AHELO Feasibility Study Report. Volume 1 - Design and Implementation. Organization for Economic Co-operation and development (OECD). Retrieved from http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf

Valanides, N. and Angeli, C. 2005. Effects of instruction on changes in epistemological beliefs. Contemporary Educational Psychology 30, 314-330.

West, E. J. (2004). Perry’s Legacy: Models of Epistemological Development. Journal of Adult Development 11, 61-70.

Footnotes

1 King and Kitchener (2004) do not call the lowest level of reflective thinking realism. However, in their model they maintain that, at the most limited level of thinking, knowledge is certain and is obtained from direct observation (p.7). This position fits metaphysical realism.