Habits of Mind and Performance Task Achievement

A study of the relationship between ‘Habits of Mind’ and ‘Performance Task’ achievement in an International School in

South-east Asia

A research project submitted for the Master of Arts in Learning and Teaching

Department of Education

University of Roehampton London

1

Habits of Mind and Performance Task Achievement

Abstract

The trend in global education is moving away from content acquisition and traditional exams, and towards ‘21st Century’ cognitive skills and performance assessments (PAs), where students are required to transfer knowledge, skills and understandings to solve real-world problems. There is a large body of theoretical literature which indicates that these ‘21st Century’ skills should be beneficial for student learning; however, there is a paucity of empirical evidence on the matter. The question remains is “Is there empirical data that indicates that development of

21st Century skills improves achievement in performance assessments?”

This study focuses on a K-12 international school in South-east Asia which uses Understanding by Design (UbD) as a framework for learning and teaching, and has adopted Costa and Kallick’s

Habits of Mind (HoMs) as a set of ‘21st Century’ skills. The research utilised a quantitative correlational design in an attempt to determine whether there is a correlation between the HoMs and student achievement in PAs. Assessment data from 354 students in middle school social studies and 246 students in high school English Language Arts was collected. The data was analysed using multiple linear regression (MLR) with the goal of determining the extent to which achievement in the HoMs affects achievement in PAs, compared to knowledge acquisition, and the development of understandings of the big conceptual ideas of a unit.

The major challenges for the study were ensuring that the data collected genuinely measured what it claimed, and the elimination of bias due to the researcher’s vested interest in the research. The former was tackled by rigorous vetting of assessment criteria and collaborative calibration of grading, whilst the latter was mitigated by the use of a transparent reflective journal throughout all stages of the investigation.

The study found a high level of correlation between the HoMs and PA achievement, although more so in middle school social studies than in high school ELA. The reasons behind the findings are discussed, and recommendations are made for implementation, and for further research.

2

Habits of Mind and Performance Task Achievement

Contents

Acknowledgements....................................................................................................................................5

Chapter 1 : Introduction, Organisational Context, and Research Objectives....................................6

Chapter 2 : Critical Literature review.....................................................................................................10

Theoretical Literature...........................................................................................................................11

The Nature of Intelligence................................................................................................................11

Cognitive Learning Theories...........................................................................................................11

Constructivism & Social Learning Theories..................................................................................13

Brain Research..................................................................................................................................14

Importance across the Domains.....................................................................................................15

Perceptions on HoM Implementation................................................................................................15

Empirical Research on Critical Thinking and Metacognition.........................................................17

Conclusions...........................................................................................................................................20

Chapter 3 : Research Methodology.......................................................................................................21

Research Question, Hypothesis and Null Hypothesis....................................................................21

Conceptual Framework........................................................................................................................21

Research Methodology........................................................................................................................23

Ethical Considerations.........................................................................................................................24

Data Collection Sources......................................................................................................................25

Validity & Reliability..............................................................................................................................27

Reliability............................................................................................................................................27

Internal Validity..................................................................................................................................29

External Validity.................................................................................................................................34

Chapter 4 : Findings, Analysis, Limitations and Evaluation...............................................................35

Findings..................................................................................................................................................35

Findings for Middle School Social Studies....................................................................................35

Findings for High School ELA.........................................................................................................42

Analysis..................................................................................................................................................46

Limitations..............................................................................................................................................48

Evaluation..............................................................................................................................................49

3

Habits of Mind and Performance Task Achievement

Chapter 5 : Conclusions and Recommendations................................................................................51

Conclusions...........................................................................................................................................51

Recommendations for Educators.......................................................................................................53

Recommendations for Further Research..........................................................................................54

Glossary of Terms....................................................................................................................................56

References................................................................................................................................................57

Appendices................................................................................................................................................63

Appendix 1: School-wide Learner Outcomes...................................................................................63

Appendix 2: Approved Ethics Response Form................................................................................64

Appendix 3: Authorisation Letter from the Board of Directors.......................................................75

Appendix 4: Essential Questions Summary Sheet..........................................................................76

Appendix 5: Essential Questions Discussion Rubric......................................................................78

Appendix 6: Habits of Mind Journal and Rubric..............................................................................80

Appendix 7: EHS1S2U1 PT – Perceptions of Social Media +Rubric...........................................82

Appendix 8: SS6S2U1 PT – Population Demographics + Rubric.................................................85

Appendix 9: SS7S2U1 PT – Land Use Conflict in the Amazon + Rubric....................................88

Appendix 10: SS8S2U1 PT – Human Rights Debate + Rubric.....................................................92

Appendix 11: EHSFS2I1 PT – In Memory (Eulogy) + Rubric........................................................96

Appendix 12: EHS2S2U1 PT – Psychological Diagnosis + Rubric............................................100

Appendix 13: EHS3S2U1 PT - Allegorical Narrative + Rubric....................................................104

Table of Figures

Figure 3.1 Conceptual Framework........................................................................................................24

Figure 4.1 Middle School Social Studies PT vs HoM Scatterplot.....................................................36

Figure 4.2 Middle School Social Studies PT vs EQs Scatterplot......................................................36

Figure 4.3 Middle School Social Studies PT vs K&S Scatterplot......................................................37

Figure 4.4 Middle School Social Studies Model Summary................................................................37

Figure 4.5 Middle School Social Studies Coefficients & Collinearity................................................38

Figure 4.6 Middle School Social Studies Residuals Statistics & Normality of Residuals..............40

Figure 4.7 Middle School Social Studies Homoscedasticity..............................................................41

Figure 4.8 High School ELA PT vs HoMs Scatterplot........................................................................42

Figure 4.9 High School ELA PT vs EQs Scatterplot...........................................................................42

Figure 4.10 High School ELA PT vs K&S.............................................................................................43

Figure 4.11 High School ELA Model Summary...................................................................................43

4

Habits of Mind and Performance Task Achievement

Figure 4.12 High School ELA Coefficients & Collinearity...................................................................44

Figure 4.13 High School ELA Residuals Statistics & Normality of Residuals.................................45

Figure 4.14 High School ELA Homoscedasticity.................................................................................46

Acknowledgements

I would like to say huge thank you to both Bena Kallick (co-author of the Habits of Mind) and Jay

McTighe (co-author of Understanding by Design) for being a great inspiration for my work in this

area.

In addition, acknowledgement is given to Katie Marquardt (Middle School Social Studies Subject

Coordinator), Emma Rochelle Carr-Gardner (High School English Language Arts Subject Coordinator), Daryl Thompson (Middle School Level Coordinator) and Norman Still (High School Level Coordinator) for their hard work and support throughout the project.

Finally, credit is due to all of the middle school social studies and high school English language arts teachers at the project school, for their part in moving education forward to meet the needs of learners in the 21st Century.

5

Habits of Mind and Performance Task Achievement

Chapter 1 : Introduction, Organisational Context, and Research

Objectives

It is abundantly evident that the world is changing rapidly, and that the educational landscape must also change if we are to prepare young people to be successful in the challenges that they will face. There are three main factors which are agents of this necessary change. Firstly, the sheer amount of information which is available to people through advances in technology, particularly the internet. Secondly, the fact that computers, machines and robots are already taking the place of humans in many jobs. Thirdly, as a consequence of the former two factors, in schooling we are attempting to prepare students for jobs which do not yet exist (Voogt and Roblin, 2012). The combination of these factors means that educational models are shifting toward a focus on transferable cognitive skills and performance assessments (PAs), where students are required to transfer knowledge, skills, and understandings to solve real-world problems.

A fascinating fact to support the first point is that between 2003 and 2009, there was an increase in the amount of information available online of 10,000%, and if the amount of information available in 2009 were to be published in physical books, it would stretch thirteen times the distance between the Earth and Pluto (Infowhelm: Global Digital Citizen, 2013). This means that a person can no longer be an ‘expert’ in anything (in the knowledge sense), and that acquisition and recall of knowledge is largely obsolete, as we can ‘Google’ it to acquire any information which we need. This naturally means that educational systems must shift away from models where students memorise facts in order to recall them for traditional examinations. Due to technological advances, 21st Century education must prepare students by lessening the emphasis on simple procedures, and instead use them as a foundation for mastery of extended complex performances, which will be requited in the future workplace (Dede, 2010).

6

Habits of Mind and Performance Task Achievement

Although there is some debate as to which cognitive skills are essential for 21st Century success, much research has been carried out in this domain (Donovan, Green and Mason, 2014; Kereluik et al., 2013), and they are generally articulated as “creativity & innovation, critical thinking & problem solving, and communication & collaboration” (Partnership for 21st Century Learning, 2007: 2). Although schools appear to agree on what changes need to occur (the intended curriculum), they are often unsure as to how to implement new practices in the classroom (the implemented curriculum), and finally how to determine if the goals have been met (the attained curriculum) (Voogt and Roblin, 2012). The educational researchers and writers, Art Costa and Bena Kallick have carried out much work in this area. They have formulated the 16 Habits of Mind (HoMs) (Costa and Kallick, 2008), which are a set of 16 cognitive dispositions which research indicates are necessary to create self-directed and life-long learners prepared for the challenges of the modern world.

If we are to prepare students for as yet unknown vocations, where they are required to solve problems and perform complex mental processes, then it stands to reason that assessment methods in schools should reflect this change (Center for Collaborative Education, CCE, 2017). For the past 20 years, the Understanding by Design Framework (Wiggins and McTighe, 2005) has supported the importance of both the shift towards conceptual understandings and authentic performance assessments. Within the Understanding by Design (UbD) framework, desired unit outcomes are categorised as knowledge and skills (acquisition - A), understandings (meaning-making - M) and transfer goals (T). In this framework, the goal is for students independently to transfer their learning to new situations, and the most important form of assessment is through authentic performance tasks.

Although there is a wealth of research (Campbell, 2006; Edwards, 2014) which indicates that students’ attainment of 21st Century Skills, and in particular the HoMs should in theory increase their achievement in performance assessments, there is a lack of empirical evidence on the

7

Habits of Mind and Performance Task Achievement

matter. The real crux of the topic is whether or not research can indicate that students’ internalisation and habitualisation of these cognitive dispositions enhances their performance in authentic assessment. This leads to the research question “To what extent does student performance in Habits of Mind assessment account for variance in performance task achievement?” A progressive international school in South-east Asia was chosen in which to conduct the study for a number of reasons. The school is accredited by the Western Association of Schools and Colleges (WASC) (Accrediting Commission for Schools, 2016), which ensures academic integrity and quality of instruction. Current educational standards have been adopted including the Common Core State Standards (CCSS) for ELA, and American Education Reaches Out (AERO) for social studies, which were the domains investigated. In addition, all of the units of study are designed using the UbD framework, and furthermore the school has not only adopted the HoMs, but has included them in the desired unit goals, links learning activities to them, and has designed and implemented robust summative assessments to measure students attainment of them. Knowledge and Skills (K&S) assessments are used to measure students’ acquisition of key skills and processes. Contributions to Essential Question (EQ) discussions are used to assess students’ development of conceptual understandings and abstract ideas. Finally, HoM summative assessments (in the form of a self-assessment, reflection, and goal-setting journal) are utilised in order to ascertain students’ abilities to identify, apply, and reflect on the HoMs. These assessments are in turn used to ascertain students’ readiness to tackle authentic performance tasks (PTs). This study is of high significance due to the large amount of secondary achievement data related to performance in the HoMs and PTs in the project school, enabling a large-scale quantitative correlational study to be carried out. Furthermore, this is the first empirical study of its kind which investigates the impact of the

HoMs on students’ performance in authentic transfer tasks.

8

Habits of Mind and Performance Task Achievement

The purpose of the study was to attempt to ascertain if there is a relationship between students’ habitualisation of the HoMs and their achievement in performance assessments, and also to ascertain whether acquisition of knowledge and skills, development of abstract understandings, or attainment of the HoMs has a greater effect on performance task achievement.

9

Habits of Mind and Performance Task Achievement

Chapter 2 : Critical Literature review

The 16 Habits of Mind (HoMs) are a collection of cognitive dispositions which the authors assert embody “having a disposition toward behaving intelligently when confronted with problems” (Costa and Kallick, 2000:1). The full list is Persisting, Managing Impulsivity, Listening with Understanding and Empathy, Thinking Flexibly, Thinking About our Thinking, Striving for Accuracy, Questioning and Problem Posing, Applying Past Knowledge, Thinking and Communicating with Clarity and Precision, Gathering Data through All Senses, Creating, Imagining and Innovating, Responding with Wonderment and Awe, Taking Responsible Risks, Finding Humour, Thinking Interdependently, and Remaining Open to Continuous Learning. The theory is that if a student values these patterns of thinking, can determine in which situation they might be useful, can skilfully employ them, and strives to reflect and improve on them, then this will lead to success in learning, employment and life in general.

Even though the Habits of Mind (HoMs) (Costa and Kallick, 2008) have been with us for almost 30 years, albeit beginning with a list of 12 rather than 16 (Costa, 1991), research into the importance of which cognitive functions are important goes back much further (Feuerstein, 1980). It could even be argued that this quest began with the classical philosophers Socrates, Plato and Aristotle (Bransford, Brown and Cocking, 2000); however, there is still some debate as to whether the HoMs are supported by a sufficient amount of academic research (Campbell, 2006). When critically appraising the available literature, it is important to investigate the theoretical research underpinning the formation of the HoMs, both educators’ and students’ perceptions of the benefits of their implementation, and any empirical evidence of their positive effect on student learning. Thus this critical literature review is organized accordingly.

10

Habits of Mind and Performance Task Achievement

Theoretical Literature

In his paper Theorising the Habits of Mind as a Framework for Learning, Campbell (2006) does an excellent job of relating the HOMs to “theories on the nature of intelligence, cognitive learning theories, social learning theories and brain research” (p.4).

The Nature of Intelligence

In essence, in the section on the nature of intelligence, Campbell draws on a multitude of previous researchers’ assertions (Beyer, 1998; Perkins, 1995; Langer, 1989; Sternberg, 1985; Whimbey, 1975; Dewey, 1933) that intelligence, rather than a single mental ability, is a complex combination of thinking skills which can be learned and practiced, including the ability to reflect, draw on experiences, create and revise goals, and adjust to situations. The argument can be made that these cognitive abilities are synonymous with the HoMs of Metacognition, Applying Past Knowledge, Remaining open to Continuous Learning and Thinking Flexibly.

Cognitive Learning Theories

The cognitive learning theories which inform the HoMs can roughly be categorized into information processing models, metacognitive models, cognitive styles, and constructivism (Campbell, 2006).

Researchers have theorized about the ways in which we process information. Schneider et al. (2003) posit that all information initially is received into the sensory register before we attend to it and move it to other storage compartments. Others have conjectured about ‘working memory’

(Baddeley, 1986) and different types of ‘long-term memory’ (Tulving, 1985). Further connectionist models (Ellis and Humphreys, 1999; McClelland, Rumelhart and Hinton, 1986) assert that the brain is a complex interconnected series of storage compartments between which information moves. For the purposes of this literature review, it is not necessary to delve

11

Habits of Mind and Performance Task Achievement

in to these theories in any great detail, it simply suffices to note that these information processing models have informed the HoMs of Gathering Data through All Senses, Applying Past Knowledge to New Situations and Thinking and Communicating with Clarity and Precision.

One can state this simply by asserting that when information is received into the sensory register, we apply Gathering Data through All Senses, when we move information from the various storage compartments, we utilize Applying Past Knowledge to New Situations and then in turn we apply Thinking and Communicating with Clarity and Precision to organize the information and use it in meaningful ways.

If we take a look at the 16 HoMs, it can be seen that the HoM Metacognition is essentially a pre-cursor to the other 15. If one consciously thinks about one’s own thinking, then one can determine which of the other HoMs need to be employed for success in a particular task. Researchers have broadly categorized Metacognition into self-monitoring and self-regulation thinking processes (Schneider and Bjorklund, 1998; Nelson and Narens, 1994). Another way of categorizing Metacognition is into planning, monitoring and evaluation processes (Pintrich, 2000; Pintrich and De Groot, 1990). The authors of the HoMs claim that a major goal of the adoption of the HoMs is to create self-directed, life-long learners (Costa and Kallick, 2008). It is clear that to achieve this, learners need to be able to self-monitor and self-regulate. It can be seen later in this review that in addition to theoretical models, there is also empirical evidence which indicates that learners who are trained in, and demonstrate metacognitive abilities have increased performance in assessments.

Cognitive learning theories (Bouckenooghe et al., 2016; Sternberg, 2001; Singh, 2017) also support usage of the HoMs; for example, Managing Impulsivity and Striving for Accuracy could aid learners who lack the ability to sit back, reflect, and carefully check their work. Questioning and Posing Problems could aid students who have a propensity to passively receive information

12

Habits of Mind and Performance Task Achievement

rather than critically appraising it for validity and reliability, and Taking Responsible Risks could be beneficial for students who are too conservative in their approaches to learning; for example, those who do not search for information outside of course resources, or are reluctant to try new ways of organizing their writing. In addition, Gathering Data through all Senses could help learners who are strongly on one side of the visual/verbal spectrum (Felder and Soloman, n.d.) . This brings up an interesting point, which whilst outside of the scope of this paper, is one which is worth mentioning. Not to mention the fact that many educators confuse Multiple Intelligences (Gardner, 2011) with learning preferences (Gardner, 1995), and the fact that research indicates that attempting to match instruction with preferences is not supported by evidence (Pashler et al., 2008), there appears to be a trend for teachers to focus on the preference where the learner is already strong, where in fact perhaps the focus should be to develop learners where they need the most assistance. In this sense, the HoMs could be a useful set of tools to achieve just that.

Constructivism & Social Learning Theories

If we take the broad definition of constructivist theory as “learning takes place when new information is built into and added onto an individual’s current structure of knowledge, understanding and skills. We learn best when we actively construct our own understanding” (Pritchard, 2009: 17), then it logically follows that the HoMs Metacognition, Thinking Interdependently, Questioning and Posing Problems, and Applying Past Knowledge to New Situations should all be attributes which aid learning. The work of the researchers Lave and Wenger (1991) asserts that learning occurs best when students can relate content to their own lives, which lends itself to the integration of Applying Past Knowledge to New Situations.

In addition, the HoMs align well with social learning theories. Social learning theory states that

“learners use observation, language and self-talk to make sense of the world and assist in their

13

Habits of Mind and Performance Task Achievement

choice of behaviours” (Campbell, 2006: 9). Furthermore, the concept of emotional intelligence has gained prominence in educational theory in recent years. We can use the definition of emotional intelligence as the ability to recognise the importance of emotions in ourselves and others, and to use these understandings to solve problems (Mayer, Caruso and Salovey, 1999). Traits which have been linked with emotional intelligence are the abilities to empathise, control impulses, and persist (Goleman, 1996), which once again link well to the HoMs. The dispositions of Metacognition, Managing Impulsivity, Listening with Understanding and Empathy, Finding Humour, Persisting, Thinking Interdependently, and Responding with Wonderment an Awe all incorporate the significance of emotions within the learning process, and are therefore supported by theories of social learning and emotional intelligence.

Brain Research

Advances in neuroscientific research should also be considered when evaluating the credibility of the HoMs, and there are four key developments which we should take into consideration. Firstly, that the brain has plasticity and intelligence is not fixed but can be learned (Zull, 2004). This lends credence to the HoMs Remaining Open to Continuous Learning and Taking Responsible Risks. Secondly, learning can cause physical changes in the brain by making new and stronger connections (Jensen, 2005). This suggests that a learning environment should be rich with stimuli, which supports Gathering Data Through all Senses. Thirdly, learning is maximized when we can make connections to prior experiences (Hardiman, 2010), which aligns with Applying Past Knowledge to New Situations. Lastly, the fact that our thoughts and emotions are linked to physical changes in our bodies through brain chemicals (Zull, 2004). When we experience physical changes, these in turn send messages back to our brains which affect our learning. Therefore, if we can manage our own thoughts and emotions (Metacognition), and understand those of others (Listening with Understanding and Empathy), then it is likely that learning will be maximized.

14

Habits of Mind and Performance Task Achievement

Importance across the Domains

There is also an abundance of literature which suggests that leaders in the field across a number of domains support the idea of the HoMs being integrated in curricula. Edwards (2014) asserts that the HoMs are necessary for success in a number of subjects. In engineering,

“systems thinking, creativity, optimism, collaboration, communication, and attention to ethical considerations” (Loveland and Dunn, 2014, cited in Edwards 2014: 17 ) are listed as essential habits. In mathematics, “creativity, work ethic, thinking interdependently, critical thinking, lifelong learning, and curiosity” (Charbonneau et al., 2009, cited in Edwards 2014: 17) are listed as indispensable for mathematical success. In science “curiosity, honesty, openness, and skepticism must also be nurtured, modeled, and practiced continuously in science classrooms at all levels until they become deeply entrenched and respected” (Liftig, 2009, cited in Edwards 2014:18). Sullivan (2012) declares that the HoMs are more important for university readiness than standardized test scores and even academic writing skills.

After reviewing the literature, it is my assertion that the HoMs are built upon a very strong foundation of theoretical research. The next question is whether or not stakeholders in schools, where the HoMs have been implemented, perceive them to be an effective set of tools to aid learning.

Perceptions on HoM Implementation

Now it has been established that there is a substantial theoretical basis upon which to base adoption, the next question that arises is if there is any research literature which indicates the HoMs effectiveness in terms of student achievement. More importantly, what really matters for educators is whether a focus on the HoMs is worth the undoubted time and effort involved in

15

Habits of Mind and Performance Task Achievement

implementation, and time spent in the classroom. Here we find that unfortunately the research is somewhat deficient.

Several studies research the perceptions of stakeholders as to whether the HoMs increase learning (Lesperance, n.d.; Charbonneau et al., 2009; Vollrath, 2016; Osman, 2016). Although in general, the perceptions were positive, these investigations have a number of flaws.

Lesperance’s work included very broad and unfocused research questions which did not appear synonymous with the data collection methodology. An ethnographic approach based on interviews and observations was used to ascertain classroom effectiveness after teachers were trained on the HoMs. The sample size of nine teachers was small, and there was a risk of obsequiousness bias, as the teachers most likely felt that they should have improved their behaviour after having received the training. However, the study does indicate improvements in both teacher perception and observed behaviour in the posttest results. Unfortunately students’ perceptions and behaviours were not considered. The researcher has acknowledged the limitations of the investigation, stating: “Further research should be conducted on student outcomes such as assessing the learning styles of students, assessing the change in thinking after the Habits of Mind are learned and practiced, and assessing how students perceive the Habits of Mind” (Lesperance, n.d. : 22). Similarly, Charbonneau’s (2009) study, which investigated dispositions synonymous with the HoMs implemented by teachers in the Department of Mathematical Sciences at the United States Military Academy at West Point, found that teachers’ perceptions of students’ thinking improved, but unfortunately there was no quantitative analysis. Likewise, the research by Vollrath (2016), which investigated the perceptions of both students and teachers using a phenomenological approach in a special needs environment, utilized self-assessment and interviews to determine if positive benefits were perceived. Yet again, there was no investigation into actual student achievement. Lastly, the investigation by Osman (2016) used experimental methodology with a group who received

16

Habits of Mind and Performance Task Achievement

coaching in the HoMs and a control group who did not, of first-year university physical education teachers. The medium to ascertain effectiveness this time was positive-thinking tests, rather than any investigation into student achievement. The experimental group perceived that they benefitted from the training.

Do these studies really tell us anything about the value of the HoMs in terms of student achievement? These investigations all conclude that if the HoMs are included in the curriculum, then the perception is that students display them. When one considers this, it appears rather obvious that this would be the case, but it tells us nothing about the value of the HoMs. To take a hypothetical case, if the HoM Ignoring Others’ Opinions (which is clearly fictitious) were to be modelled by teachers and students were required to apply it, then it could be assumed that students would become rather good at doing so. Of course, this HoM is negative, and would most likely have a negative effect on students’ attainment of desired learning outcomes. What we are attempting to ascertain is whether or not embedding the HoMs into our learning and teaching actually has a positive effect on students’ attainment of 21st Century curricular goals, and specifically their ability to solve real-world problems in authentic performance assessments. This leads to the next part of the review, which looks at whether there is any empirical evidence that student’s display of the HoMs improves their performance in assessments.

Empirical Research on Critical Thinking and Metacognition

At the time of writing, there are no empirical studies which attempt to determine a correlation between HoM performance and any kind of assessment achievement. Furthermore, no investigations have been conducted which investigate HoM implementation and student achievement. These are important distinctions, as even if the HoMs have been implemented, and even if there is a perception that they have impacted learning, this still really tells us nothing about whether application of the HoMs has a positive influence on achievement.

17

Habits of Mind and Performance Task Achievement

There have, however, been numerous recent studies which have attempted to determine if there is a correlation between metacognitive strategies and student performance in mathematics (Cornoldi et al., 2015; Zakaria, Yazid and Ahmad, 2009; Pennequin et al., 2010; Onu et al., 2012). These studies are highly relevant to the research question, as metacognitive skills are at the very heart of the HoMs. According to Costa and Kallick, Thinking about Thinking (Metacognition) is "our ability to know what we know and what we don't know. It is our ability to plan a strategy for producing the information that is needed, to be conscious of our own steps and strategies during the act of problem solving, and to reflect on and evaluate the productiveness of our own thinking" (2008: Chapter 2). It is clear that in many ways,

Metacognition is the precursor to all of the other HoMs, as students will employ it to evaluate their progress during task completion, and also to choose which of the HoMs are needed for success. The study by Zakaria, Yazid and Ahmad (2009) is of particular interest. The researchers investigated 378 pre-university students in Malaysia, and attempted to determine a correlation between metacognitive awareness and students’ performance on mathematical problem-solving assessments. The study used a Metacognitive Awareness Questionnaire (MAQ) which was modified based on that developed by the researchers Schraw and Dennison (1994) to determine students’ metacognitive skills, and a Mathematical Problem Solving Test (MPST) including probability topics to ascertain students’ mathematical problem solving abilities. Pearson’s correlation coefficient was used to determine the relationship between metacognitive awareness and performance, revealing a significant positive correlation. In other words, the higher the students’ metacognitive awareness, the higher their achievement in the test. This study is highly relevant to the research question, as the assessments involved problem solving, although it is not clear how authentic the problems in the test were.

Another fascinating study is that carried out by Cornoldi et al. (2015). The research involved 135 fourth and fifth-grade students at a school in Northern Italy using an experimental design, where

18

Habits of Mind and Performance Task Achievement

the experimental group received training on both metacognition and working memory (WM), and the control group received the training at a later date to avoid ethical issues. This study is also particularly relevant to the research question, as all participants were assessed on both the two aspects involved in the training, which is metacognitive beliefs about math and WM updating capacity, and arithmetical problem-solving ability both pre and post-training. Furthermore, regression analysis was employed in an attempt to determine which gains had the most effect on problem-solving performance. The study also outlines the content of both the metacognitive and WM sessions, including activities such as listening to stories and then recalling information whilst being asked to consider the importance of working memory in problem solving, which could be of use to educators who wish to implement similar programmes. The results of the study showed not only statistically significant gains in metacognition and WM, but also in the

‘transfer’ to the arithmetical problem solving task after the training. However, when regression analysis was applied, the only significant predictor for problem solving from the training was the gain in WM. The researchers conclude that the results support the use of this kind of training on metacognition and WM in mathematics programmes, particularly for struggling learners. The conclusions support the hypothesis of this study that training in the HoMs (particularly

Metacognition) should in theory result in gains in problem-solving performance, at least in mathematics.

There are two further relevant quantitative studies (Pennequin et al., 2010; Onu et al., 2012) which investigated whether training in metacognitive strategies positively affect students’ performance in mathematics courses. The former used an experimental approach and revealed that the experimental group which had the training significantly outperformed the control group in problem solving posttests. The study by Onu et al. (2012) used a similar approach and also returned positive results, although the nature of the posttests is unclear, and some of the

19

Habits of Mind and Performance Task Achievement

strategies, such as the use of acronyms to aid memory of processes are questionably metacognitive.

Although ‘critical thinking’ is not included in the 16 HoMs, Questioning and Problem Posing is certainly a part of the more general set of critical thinking skills, as students need to formulate questions to determine validity, credibility and reliability of sources, and Listening with Understanding and Empathy is needed in order to critically ascertain bias and purpose of information. In a paper by O’Hare and McGuinness (2015), the authors hypothesized that critical thinking tests would be a stronger predictor than Cambridge ‘A’ level results of success in degree programmes. The study found that ‘A’ levels were the stronger predictor in the first year, but the critical thinking tests surpassed their predictability at the end of the third year, in particular the skill of making inferences. The study used sound methodology, although the degree achievement was limited to one university psychology course. This study is relevant as one of the purposes of this paper is to help determine what the focus of learning activities should be, in relation to how far the goals can predict future success of students.

Conclusions

The conclusions of this critical literature review are that the HoMs are solidly grounded in an abundance of theoretical research, there is a perception amongst stakeholders of the positive impact of HoM implementation on student learning, and that there is some strong empirical evidence of gains in mathematical problem solving when training in metacognitive strategies is administered. However, there is a gap in the academic literature with regards to the specific effect of students’ demonstration of the HoMs on authentic performance task achievement, which validates the purpose of this research.

20

Habits of Mind and Performance Task Achievement

Chapter 3 : Research Methodology

Research Question, Hypothesis and Null Hypothesis

Research Question:

“To what extent does student performance in Habits of Mind assessment account for variance in performance task achievement?”

Hypothesis:

An increase in performance in Habits of Mind assessment will result in an increase in performance task achievement.

Null Hypothesis:

There is no difference in performance task achievement when Habits of Mind performance varies.

Conceptual Framework

It is first important to articulate the conceptual framework upon which the study is based. The school in question uses Understanding by Design (UbD) (Wiggins and McTighe, 2005) as a framework for the curriculum. UbD is essentially three things. Firstly it is a framework for curriculum planning, secondly it is a way of thinking about learning and teaching, and thirdly it is a set of useful tools and templates to facilitate planning and teaching (Hawker Brownlow Education, 2013). Stage 1 of UbD is the ‘desired results’ section. Within UbD, the desired learning outcomes are broken down into three distinct sections, ‘Acquisition’ (A),

‘Understanding’ (U) and ‘Transfer’ (T)1. The ‘Acquisition’ section designates the essential knowledge and skills that students will be required to know, and be able to do by the end of the unit of study. ‘Knowledge’ in this context refers to facts, definitions. and formulae which can be

1 A Glossary of Terms has been provided to explain all acronyms used in more detail.

21

Habits of Mind and Performance Task Achievement

assessed via recall questions. Examples within the social studies context could be knowledge of major historical events, knowledge about the key ‘players’ within an era, and definitions of key academic vocabulary such as ‘propaganda’ and ‘censorship’. ‘Skills’ within this section refers to the set of physical or cognitive abilities which require acquisition and practice, and can be assessed in isolation. Within the social studies domain, examples of skills are the ability to determine cause and effect, the capability of ascertaining bias in a primary source, and the skill of constructing a timeline to represent chronology.

The ‘Understandings’ section refers to abstract conceptual understandings about big ideas within a unit. The understandings are paired with Essential Questions (EQs) which are overarching, open-ended questions designed to promote inquiry, discussion and debate around the big ideas. Examples of Essential Questions are “Is there a best form of government?” , “How can we deal with scarcity?”, and “How can we mitigate conflict and misunderstandings?” An example of an understanding which may stem from the first EQ is “Students will understand that different types of government have been implemented throughout different times and locations throughout history, and have differing strengths and weaknesses in terms of their ability to serve the needs of all people.” These abstract understandings will develop and deepen over time, both within and across units, and can be transferred to new and different situations.

Author Jay McTighe describes ‘Transfer’ as “effective uses of understanding, knowledge, and skill that we seek in the long run; i.e., what we want students to be able to do when they confront new challenges – both in and outside of school” and “Transfer is about intelligently and effectively drawing from your repertoire, independently, to handle new contexts on your own” (McTighe, 2014: 1). Adopters of UbD are encouraged to create a set of ‘Long-term Transfer

Goals’ which are a list of overarching aims, which students should be able to independently perform without the assistance of a teacher or other adult. These are synonymous with the concept of ‘School-wide Learner Outcomes’ (SLOs), which are a part of the ACS WASC Focus

22

Habits of Mind and Performance Task Achievement

on Learning protocol (Accrediting Commission for Schools, Western Association of Schools and Colleges, 2014). As an ACS WASC accredited institution, the school chosen for the study has articulated a set of twelve overarching SLOs (Appendix 1: School-wide Learner Outcomes), a selection of which populate the ‘Transfer’ section of the UbD unit planning template and drive all curriculum planning. Jay McTighe also asserts that “Transfer calls for the use of habits of mind; i.e., good judgment, self-regulation, persistence along with academic understanding, knowledge and skill” (McTighe, 2014: 1) and therefore the school involved in the investigation has added a section for the HoMs in Stage 1 of the UbD design process, so that these cognitive habits are also made explicit, and are taught and assessed along with the knowledge, skills, understandings and transfer.

Research Methodology

A correlational quantitative study was carried out using Multiple Linear Regression (MLR) as the design strategy. Utilising MLR, a researcher can ascertain which independent variables (IVs) can account for the most and least variance in the dependent variable (DV) (Punch and Oancea, 2014). As within the conceptual framework, acquisition of knowledge and skills, development of enduring understandings, and display of the HoMs are all considered necessary milestones on the way to success in the Performance Task (PT) assessment, the PT assessment was designated as the DV, and the other three assessments as the IVs. In other words, the research attempted to determine whether a students’ acquisition of knowledge and skills, attainment of abstract conceptual understandings, or display of cognitive dispositions has a greater effect on their ability to solve real-world problems through PT assessment. The conceptual framework can be illustrated as follows:

23

Performance

Figure 3.1 Conceptual Framework

Pearson’s correlation coefficient was used to determine the strength and direction of any relationship between the IVs and DV (Field, 2009). Firstly, the squared multiple correlation coefficient, R2 was estimated which gave an estimate of how much variance in PT achievement can be accounted for by variance in the other three forms of summative assessment. Secondly, standardized partial regression coefficients (beta weights) were attached to the IVs in order to indicate how important each of the three assessment types were in predicting the PT score. In this way, the beta weight for the HoM assesment told us how much of a change we would expect to see in PT achievement for one unit of change in the HoM score, if all of the other independent variables were kept constant.

Ethical Considerations

With regards to ethical considerations, the data needed for the research was categorized as secondary data, as all the student achievement data is stored for grade book reporting purposes in the school’s Student Information System (SIS), and would have been so irrespective of the investigation (University of Surrey, n.d.) . Furthermore, no students’ personal data was required as no disaggregation of data was performed based on student demographics; therefore all data

24

Habits of Mind and Performance Task Achievement

was completely anonymised, and no consent was needed to be obtained from students or parents. (University of Roehampton, London, 2014) (Appendix 2: Approved Ethics Response Form). The consent of the school owners to perform the research was obtained via a signed official letter (Appendix 3: Authorisation Letter from the Board of Directors).

To address ethical considerations regarding that of potential bias, subjectivity and a vested interest in the results (Punch and Oancea, 2014), two major steps were taken. Firstly, a reflective research journal was kept documenting all of the events throughout the project, including the thoughts and decisions made. This journal had particular repercussions with regards to the validity and reliability of the data, as can been seen in a later section of the paper. In this way, it can be claimed that the researcher’s inside knowledge increased the objectivity of the study, as it was documented how personal interest was separated from an objective analysis of the data (Ortlipp, 2008). Secondly, as the primary researcher has an in-depth knowledge of the processes involved in collecting the data and a stake in the analysis, the Middle School Social Studies Subject Coordinator, the High School English Language Arts Subject Coordinator, the Middle School Level Coordinator, and the High School Level Coordinator were recruited as ‘critical friends’ to check the data collection, analysis, results, and conclusions to eliminate bias.

Data Collection Sources

The school in question uses four summative assessments within each unit to measure student achievement of the three independent variables and the dependent variable, and all of the student achievement data required was exported from the school’s SIS, and subsequently uploaded to the Statistical Package for the Social Sciences (SPSS) for analysis. Knowledge and Skills (K&S) assessments are used to measure students’ acquisition of the essential knowledge and skills required for success in the PT. The K&S variable is assessed through multiple choice,

25

Habits of Mind and Performance Task Achievement

matching, and short response questions, and an X out of Y score is given which is then converted into a percentage.

Students’ attainment of the Enduring Understandings (EU) of the unit are measured both by an analysis of discourse in online discussions in Google Classroom around the EQs, coupled with evaluation of an EQ summary sheet (Appendix 4: Essential Questions Summary Sheet), where students summarise their understandings of the big ideas, framed around the six facets of understanding Explanation, Interpretation, Empathy, Perspective, Application and Self-knowledge (Wiggins and McTighe, 2005). Students’ responses are evaluated via a rubric (Appendix 5: Essential Questions Discussion Rubric) with criteria for Frequency and Quality of Contributions, Collaboration, Critical Thinking, and one criteria for each of the Enduring Understandings, where each criteria is evaluated on a 0 to 4 scale. A score of 4 indicates expectations have been exceeded, 3 indicates that they have been met, 2 indicates that they have been approached, 1 indicates that they have been attempted, and 0 indicates no attempt has been made. A final score for the EUs is given as a percentage by first calculating the X out of Y score and then dividing the numerator by the denominator. Although it could be argued that only the criteria Enduring Understandings measures the extent to which students understand the conceptual big ideas, it was decided to include the other criteria, as engaging in debate, critiquing others’ opinions and building off others’ ideas are also instrumental in developing understandings

Students’ display of the HoMs are evaluated by the following criteria:

a) Teacher observation of student behaviour throughout the unit.

b) Student provided evidence of where they have displayed the HoMs linked to unit outcomes, and missed opportunities where they believe that their performance could have been improved by employing a particular HoM.

26

Habits of Mind and Performance Task Achievement

c)A self-assessment and goal-setting section.

(Appendix 6: Habits of Mind Journal and Rubric)

The performance descriptors are quantified on the same scale as for the EUs, an X out of Y score is calculated, and this is converted into a percentage.

PTs for each unit are developed using the GRASPS template, where G = Goal, R = Role, A = Audience, S = Situation, P = Product, and S = Specifications. The concept is to provide students with an authentic context where they are required to transfer their knowledge, skills, understandings and HoMs to a new situation in order to solve a real-world problem. Achievement in PTs is assessed using analytic rubrics ,where criteria are linked to Long-Term Transfer Goals, or SLOs (Wiggins and McTighe, 2012), academic standards, and the HoMs. In this way, each student’s ability to transfer their learning to new situations is measured using the same scale as for the EUs and HoMs. These scores are also converted to an X out of Y figure, and then into a percentage.

Validity & Reliability

Reliability

Reliability of data can be described as how consistent the measure of the quality is, if the measurements were taken at different times by the same rater (intra-rater reliability) and also by different raters (inter-rater reliability). There are two main attributes to reliability, stability and equivalence (Heale and Twycross, 2015). Stability is “the consistency of results using an instrument with repeated testing” (p.67) and equivalence is “consistency among responses of multiple users of an instrument, or among alternate forms of an instrument” (p.67). The initial plan was to collect and analyse data from Semester 1 of Academic Year 2017-2018; however, due to a number of new teachers joining the relevant departments at the beginning of the year,

27

Habits of Mind and Performance Task Achievement

the main researcher realised that more practice and professional development was required to maximise the reliability of the measures. This was noted in the reflective journal, and the decision was taken to collect data from the first unit of Semester 2, after professional development had been administered, and the teachers already had experience grading Semester 1 assessments. In addition, to attempt to maximise the reliability of the data collected in the study, teachers within a course attended grading calibration sessions where they all graded the HoM journals, EQ discussions and PTs silently and separately before sharing their scores with the rest of the group. Subsequently, any divergences were discussed and it was attempted to reach a consensus. This was carried out to attempt to ensure that the analytic rubrics were being interpreted in the same way across all raters. The main researcher also attended some of these sessions and journaled his experiences and observations. As the answers to the K&S assessments were generally graded as either correct or incorrect, it was not deemed necessary to hold grading calibration sessions for these assessments.

Whilst attending calibration sessions for the EQ discussions, two major points related to reliability were noted by the researcher. Firstly, there was an instance with the Grade 7 social studies unit, where although students were providing detailed answers framed by the six facets, it was debatable whether the EQ was being adequately addressed. One of the EQs for the unit was “Who owns what and why?” and was designed to lead to understandings regarding the mechanisms which can lead to land, resource, and wealth ownership. In their answers, some students focused on the purpose of the land ownership, i.e. the different ways land is used, rather than the ways in which the owners came to be awarded ‘ownership’ of the land. This was clarified by the researcher amongst the participating teachers. Secondly, during the same calibration sessions, it became apparent that there were some misconceptions regarding the facet of understanding application. When students demonstrate understanding through application, they “use knowledge in diverse situations and new contexts” (Mongan-Rallis, 2005).

28

Habits of Mind and Performance Task Achievement

The misconception which became apparent was that some teachers were giving credit for student examples of application of understandings by protagonists in the topical unit content, rather than examples of application by the students themselves. Again, this was clarified by the researcher to the teachers.

During the HoM calibration sessions, three major points arose. Firstly, some teachers had the tendency either to compare the same student’s different submissions with each other, or to compare different students’ submisisons in order to determine a score. It was clarified that the grading should always be ‘standards-based’ rather than ‘norm-referenced’, i.e. the submissions should be graded according to the analytic rubric rather than any comparison being employed. The second question that arose was with regards to the self-assessment and goal-setting criterion. It was unclear to some teachers the timescale that the students were required to goal-set within. It was clarified that they are asked to make a general statement about how they intend to improve their display of a particular HoM, and how they intend to measure success. Finally, the question was brought up of how much detail was required in the HoM evidence statements. In one example, the HoM was Thinking and Communicating with Clarity and Precision. A student had performed well, reflecting and giving evidence that they had added specific detail in a speech (even adding what that detail was) to improve communication and clarity. The question then arose whether the detail given was sufficient at that grade level. The researcher responded that the answer was to look at the academic standards for ELA to determine what level of complexity should be expected for supporting detail at this grade level. It became evident that the level of detail was sufficient, and credit should be given for the submission. These steps underlie the importance of reliability checks when utilising analytic rubrics for grading complex assessments.

Internal Validity

29

Habits of Mind and Performance Task Achievement

Validity can be described as “the extent to which a concept is accurately measured in a quantitative study” (Heale and Twycross, 2015: 66), or in other words, whether or not we are truly measuring what it is that we claim we are. The first point to address here is a philosophical one. When we are discussing the concepts of ‘abstract understandings’, ‘Habits of Mind performance’, and ‘Transfer’, the question is whether or not these are variables that can actually be measured. We can trace this debate back to the Italian mathematician Galileo Galilei (and probably further) whose maxim was famously “'to measure everything measurable and to make what is not measurable capable of being measured”. A more recent relevant quote is “if it happens you can count it” (Whiting, 1980, cited in Turner and Schechner, 1988: 2). The argument could be made that these three variables are not directly observable, and therefore cannot be measured. This is the problem of the so-called ‘latent variable’ within the social sciences, and Bollen (2002) does an excellent job of discussing the varying definitions of latent variables, and how they can be treated in investigations. A very useful informal definition of the latent variable is one of a data reduction device (Bollen, 2002) and that “the latent variable or factor is a convenient means of summarizing a number of variables in many fewer factors” (p.

608). We can see that abstract understandings, HoM performance and Transfer fit this description well. A student’s conceptual understanding of a big idea can be measured by the extent to which they can write about the concept through the lens of the six facets (Wiggins and McTighe, 2005). A student’s ability to display an HoM within the context of particular learning outcomes can be measured by their observable behaviours in class, their ability to provide evidence of application, and their ability to self-reflect, self-assess, and goal-set. Finally, as the school chosen for the study has articulated a set of twelve Long-term Transfer Goals (synonymous with the SLOs within the ACS WASC Focus on Learning framework), ‘Transfer’ can be measured via the use of analytic rubrics which assess the extent to which a student has combined acquired knowledge and skills, understandings, and the HoMs to demonstrate one or more of the SLOs in a real-world context.

30

Habits of Mind and Performance Task Achievement

It is also useful to describe the assumptions which have been made for the purposes of the research. There could also be debate as to whether the variables measured are categorical, continuous, or a hybrid of the two. For example, within a PT, should we be evaluating whether a student has demonstrated Transfer or not (categorical), or evaluating the extent to which the student has demonstrated Transfer (continuous)? As can be seen in the appendices, analytic rubrics have been designed to determine five different levels of performance, so therefore the assumption has been made that the variables are continuous. In addition, it is useful to mention the problems surrounding the use of latent variables in a multiple regression model, and this again has been discussed in depth by Bollen (2002) who states profoundly “we readily see that much of psychology and the social sciences routinely use such unobserved or latent variables in their statistical modelling. Hence, to purge our models of unobservable or latent variables would require that we eliminate virtually all of the statistical techniques common in the social sciences” (p. 618). Now that these debates and assumptions have been raised, it is up to the reader to determine their own standpoint on the matter, and therefore their view on the validity of the data used.

When considering the validity of the assessment tools used, it is important to consider content, construct, and criterion validity (Heale and Twycross, 2015). Firstly, content validity is “the extent to which a research instrument accurately measures all aspects of a construct” (p.66). As previously stated, content validity was a concern of the primary researcher at the beginning of the study, and caused a delay in the research as peer-review of unit plans and assessment were carried out in conjunction with professional development for teachers on interpreting and rubrics and calibrating grading. As the primary goal of the research was to determine the extent to which performance in HoM assessment accounts for variance in PT achievement, it was important for the PT to actually require demonstration of the HoMs selected. An example of poor content validity in this context would be assessment of Thinking Flexibly through the HoM

31

Habits of Mind and Performance Task Achievement

Journal and attempting to determine correlation between that measure and a PT which does not require flexible thinking for success. In this case, it could be imagined that a student may excel in her application of flexible thinking, but still perform poorly in the PT, as other HoMs were required for success, thus revealing no correlation. In fact, during the peer-review process to check internal validity, it was observed that the high school English Language Arts (ELA) unit for Level 1 (which is studied in either grade 9 or 10) (Appendix 7: EHS1S2U1 PT – Perceptions of Social Media +Rubric) was lacking to a degree in content validity. This unit is based around the novel A Picture of Dorian Gray , and in the PT students are required to write a feature article for a magazine arguing for or against the statement “The way beauty is portrayed in social media is not only a reflection of the shallow nature of society, but it is also a contributing factor.” In Stage 1 of the UbD unit plan, the SLOs “Communicate effectively for a wide variety of purposes and audiences within and across cultures” and “Convey appreciation of the arts, sciences and the beauty of the natural world” are selected, alongside the HoMs Managing Impulsivity, Questioning and Problem Posing and Finding Humor. However, when alignment was attempted between the PT assessment criteria and the SLOs and HoMs, it was revealed that only the communication SLO is assessed, and only the questioning HoM required for success. In other words, students could potentially succeed in the PT without managing their impulsivity or finding humour. In this case, content validity was determined to be weak. The decision was made to include the Level 1 unit in the research, as it could serve as a useful indicator as to whether alignment does indeed account for variance in the correlation between the HoMs and PT achievement.

The content validity of the Grades 6, 7 and 8 social studies units and the high school Foundation, Level 2 and Level 3 units was determined to be high and they were included in the study (Appendices 8-13 p.85). The issue of unit plan alignment and content validity is further

32

Habits of Mind and Performance Task Achievement

discussed in Chapter 4, and is a key point of the main recommendations made in Chapter 5 p.54.

Construct validity is “whether you can draw inferences about test scores related to the concept being studied” (Heale and Twycross, 2015: 66). Construct validity can furthermore be broken down into homogeneity, convergence and theory evidence. Homogeneity is whether or not the instrument measures one construct, and there certainly could be some debate around this point in relation to the study. For example, with the HoM Journal, students are required to self-reflect, self-assess and goal-set in addition to displaying the HoM, and to give evidence of application. With the self-reflection element, it is clear that Metacognition is additionally required to the specific HoM being measured, as it is really a precursor to all of the other HoMs as discussed in Chapter 2. In this sense, homogeneity was considered valid. With regards to Transfer, this construct is (or rather should be) a combination of the three other variables, and this is the very reason that correlation was hypothesised. Convergence is when an instrument measures concepts similar to other instruments, and as the assessments utilised are unique to the school, it was not possible to ascertain this within the scope of the research. Lastly, theory evidence is when behaviour is observed which is “similar to theoretical propositions of the construct measured in the instrument” (p.66). This would certainly be possible to measure, and should be considered for further research in this area. The last category of validity is criterion validity, which is “the extent to which a research instrument is related to other instruments that measure the same variables” (p.66). This is certainly an interesting proposition, and forms part of the recommendations in Chapter 5 p.55.

When preliminary data sets were analysed, it was decided to eliminate any data points where the value was 0. A mean score of 0 indicated that the summative assessment (or part thereof) had not been submitted. Therefore, the 0 would actually tell us nothing about whether the student had acquired the relevant knowledge and skills, attained the required understandings,

33

Habits of Mind and Performance Task Achievement

or displayed the HoMs, and thus inclusion would serve to invalidate the data. Finally, the project school has a policy of a late submission penalty, and these submissions were also excluded for the same reason.

External Validity

An experiment is deemed to have external validity when the results “are generalizable to groups, environments, and contexts outside of the experimental settings” (Onwuegbuzie, 2000: 3). Onwuegbuzie also states pertinently that, “Even if a particular finding has high internal validity, this does not mean that it can be generalized outside the study context” (p.7). Twelve threats to external validity have been identified, although in the interests of brevity, the main five which were deemed to be threats are addressed here. Population validity is deemed to be relatively high as the sample size used was random and large. However, as the school’s student population is made up largely of Cambodian students, and almost all are English Language Learners (ELLs), further research is needed across different student populations p.54.

Ecological validity is also assumed as both UbD and the HoMs are frameworks used in educational institutions globally. However, as with population validity, as no educational context is identical, further research is recommended across different schools, districts and nations p.54. Temporal validity refers to whether findings can be generalised across time, and is a threat to this study in so far as it is to almost all educational studies. Multiple-treatment interference is important as students’ general exposure to and amount of training in the HoMs could certainly affect the predictability potential of the HoMs for other forms of performance. It could be hypothesised that the more training a student has had, the more proficient she would become, and therefore more able to apply the HoMs to successfully complete diverse tasks.

Researcher bias was identified early on as a potential threat, and was addressed through the reflective journal, and peer-review of the results, analysis, and interpretation.

34

Habits of Mind and Performance Task Achievement

Chapter 4 : Findings, Analysis, Limitations and Evaluation

After the data had been cleaned of all scores which had been adjusted for late submission or which were scored as zero for non-submission, there was data for 354 students in middle school social studies across 19 classes, and 246 students in high school ELA across 18 classes. Due to a change in teacher mid-year, three classes from middle school social studies were excluded from the study due to temporal validity issues, as their knowledge and skills assessments were graded after the performance task.

The mathematical model used was as follows:

y = b₀ + b₁x₁ + b₂x₂ + b₃x₃ + ej

Where:

y is the dependent variable (PT achievement)

x₁, x₂, x₃ are the independent variables (HoM, EQ and K&S achievement)

b₀ is the intercept coefficient

b₁, b₂, b₃ are the slope coefficients

ej is the error term for the jth student

Findings

Findings for Middle School Social Studies

35

Habits of Mind and Performance Task Achievement

Figure 4.1 Middle School Social Studies PT vs HoM Scatterplot

Figure 4.2 Middle School Social Studies PT vs EQs Scatterplot

36

Habits of Mind and Performance Task Achievement

Figure 4.3 Middle School Social Studies PT vs K&S Scatterplot

For middle school social studies, the correlation between achievement in the HoM, EQ and K&S assessments and performance in the PT is clear from the scatterplots. The data shows clear

‘cigar-shaped’ patterns including some outliers where either students didn’t perform well in the milestone assessments but did so in the PT, and vice versa. The correlation is clearest for the HoMs, followed by EQs and then K&S. The scores for K&S are grouped more in the higher end of achievement, which could be expected due to the nature of the assessment requiring recall and display of basic skills.

Model Summaryb

1. Predictors: (Constant), Habits of Mind, Knowledge and Skills, Essential Questions

2. Dependent Variable: Performance Task

Figure 4.4 Middle School Social Studies Model Summary

37

Habits of Mind and Performance Task Achievement

For middle school social studies, the summary shows that the model used is strong, with an R Square figure of .492 which means that 49.2% of variation in PT performance can be explained by variation in HoM, EQ and K&S achievement. Therefore, 50.8% of the variation was due to additional independent variables, and random error.

Coefficientsa

Figure 4.5 Middle School Social Studies Coefficients & Collinearity

For middle school social studies, multiple linear regression was carried out to investigate the relationship between achievement in HoM, EQ and K&S assessments, and performance in PT assessment. The model is:

PT achievement (y) = 10.085 + 0.267*(EQ achievement) + 0.215*(K&S achievement) + 0.355 * (HoM achievement).

There was a significant relationship between all three independent variables and the dependent variable (p < 0.001). We know this as the Sig column contains the p-values for each of the

38

Habits of Mind and Performance Task Achievement

independent variables. The hypothesis being tested for each is that the coefficient (B) is 0 after controlling for the other variables. In this case, for example, the effects of Knowledge and Skills achievement and Essential Question achievement were removed before assessing the relationship between HoM achievement and Performance Task achievement. A p-value < 0.05, provides evidence that the coefficient is different to 0. The HoMs had the highest predictive power, with a 1% increase in HoM achievement resulting in a .355% increase in PT performance. This was followed by EQs, with a 1% increase in EQ achievement resulting in a

.267% increase in PT performance. Finally, a 1% increase in K&S achievement resulted in a

.215% increase in PT performance. A practical example of this would be that if a student scored 50% in all three summative assessments, their PT score would be 51.935%. If their HoM scores rose by 30% to 80%, their PT score would rise to 62.585% if the EQ and K&S scores remained the same.

In the collinearity statistics, the VIF scores for all independent variables are close to 1, which shows that multicollinearity was not a problem. In other words, the variance in the independent variables could not be accounted for by variance in the other independent variables.

Residuals Statisticsa

39

Habits of Mind and Performance Task Achievement

Figure 4.6 Middle School Social Studies Residuals Statistics & Normality of Residuals

For middle school social studies, the histogram shows that the residuals were approximately normally distributed. As residuals are elements of variation unexplained by the fitted model, the assumption is that they are roughly independently distributed. If they are not then this indicates structure in the residuals, which means that the model would need adjusting to explain this. As we can see, this is not an issue for the model used in the research.

40

Habits of Mind and Performance Task Achievement

Figure 4.7 Middle School Social Studies Homoscedasticity

For middle school social studies, there is no pattern in the scatter. The width of the scatter as predicted values increase is roughly the same so the assumption of homogeneity of variance and linearity has been met.

41

Habits of Mind and Performance Task Achievement

Findings for High School ELA

Figure 4.8 High School ELA PT vs HoMs Scatterplot

Figure 4.9 High School ELA PT vs EQs Scatterplot

42

Habits of Mind and Performance Task Achievement

Figure 4.10 High School ELA PT vs K&S

For high school ELA, the correlation between achievement in the HoM, EQ and K&S assessments and performance in the PT is clear from the scatterplots, although less so than in middle school social studies. The correlation is clearest for K&S, followed by HoMs and then EQs, which differs from middle school social studies. Possible reasons for this are discussed in the analysis.

Model Summaryb

1. Predictors: (Constant), Knowledge and Skills, Essential Questions, Habits of Mind

2. Dependent Variable: Performance Task

Figure 4.11 High School ELA Model Summary

For high school ELA, the model summary shows that the model used was moderately strong, with an R Square figure of .29 which means that 29% of variation in PT performance could be explained by variation in HoM, EQ and K&S achievement. Therefore, 71% of the variation was

43

Habits of Mind and Performance Task Achievement

due to additional independent variables, and random error. This unexplained variance is 20%

higher than in middle school social studies, which is discussed in the analysis.

Coefficientsa

a. Dependent Variable: Performance Task

Figure 4.12 High School ELA Coefficients & Collinearity

For high school ELA, multiple linear regression was carried out to investigate the relationship between achievement in HoM, EQ and K&S assessments and performance in PT. The model is:

PT achievement (y) = 36.012 + 0.11*(EQ achievement) + 0.17*(HoM achievement) + 0.212*(K&S achievement).

There was a significant relationship between K&S and PT achievement (p < 0.001), HoM and PT achievement (p = 0.001) and EQ and PT achievement (p = 0.008). The K&S had the highest predictive power, with a 1% increase in K&S achievement resulting in a .212% increase in PT performance. This was followed by HoMs, with a 1% increase in EQ achievement resulting in a

.17% increase in PT performance. Finally, a 1% increase in EQ achievement resulted in a .11%

44

Habits of Mind and Performance Task Achievement

increase in PT performance. . A practical example of this would be that if a student scored 50% in all three summative assessments, their PT score would be 60.612%. If their HoM scores rose by 30% to 80%, their PT score would rise to 65.712% if the EQ and K&S scores remained the same.

As for middle school social studies, in the collinearity statistics, the VIF scores for all independent variables are close to 1, which shows that multicollinearity was not a problem. In other words, the variance in the independent variables could not be accounted for by variance in the other independent variables.

Residuals Statisticsa

Figure 4.13 High School ELA Residuals Statistics & Normality of Residuals

45

Habits of Mind and Performance Task Achievement

For high school ELA, the histogram shows that the residuals were approximately normally distributed. Again, this is important as there does not appear to be any structure unexplained by the model.

Figure 4.14 High School ELA Homoscedasticity

For high school ELA, there is no pattern in the scatter. The width of the scatter as predicted values increase is roughly the same so the assumption of homogeneity of variance and linearity has been met.

Analysis

The findings indicate that the null hypothesis can be rejected, and also indicate a high level of correlation between achievement in the HoMs and the PT. Although the particular focus of this research is new, the findings complement both the perception data from previous studies in schools where HoMs have been implemented (Lesperance, n.d.; Charbonneau et al., 2009; Vollrath, 2016; Osman, 2016), and studies where training in metacognitive strategies improved assessment performance (Cornoldi et al., 2015; Zakaria, Yazid and Ahmad, 2009; Pennequin et al., 2010; Onu et al., 2012). Three particularly interesting points for discussion are the

46

Habits of Mind and Performance Task Achievement

differences in results between the two subjects and school level, the lack of multicollinearity for the independent variables, and the degree of success of the model itself.

The question arises as to why there is a difference in the correlations between the two subjects and school levels. It is possible that this can be explained by the internal validity issues described in Chapter 3 p.32. More specifically, it became evident during peer review that there was a higher level of alignment between Stage 1 (Desired Results) and Stage 2 (Evidence Collection) in the middle school social studies units than in the high school ELA units. This difference also supports the hypothesis in some way, as the results indicate that when the HoMs are directly required to be displayed in the performance assessment, a higher level of achievement in them will lead to a greater variance in performance. Furthermore, the higher correlation between the K&S assessment and the PT in high school ELA could be explained by the fact that the PTs in this subject were generally writing tasks, which are more skills-based (assessed on criteria such as writing organisation),which were also assessed in the K&S assessments.

The second point worthy of discussion is that the study revealed a lack of correlation between the independent variables themselves. This could be counter-intuitive to some readers, who may assume, for example, that if a student acquires knowledge and skills, then automatically his or her conceptual understandings would increase. The study does not support that theory, and although the different categories of desired results are, of course, interrelated to some degree, there is enough of a difference for them to be treated separately in planning, instruction and assessment. This does align with the concept of differentiating between Acquisition (A), Understanding (U) and Transfer (T) within the UbD framework (Wiggins and McTighe, 2005) and also leads to the recommendation that the HoMs should be included in Stage 1, and treated separately from A, M and T. This is further discussed in Chapter 5 p.54.

47

Habits of Mind and Performance Task Achievement

The aggregated predictability value from the two models was around 40%, which is by no means insignificant. However, to look at this another way, around 60% of the variability in PT achievement could not be explained by the independent variables. This leads to the question of what the other independent variables are which can account for further variance. Some assumptions were made for the study regarding variables such as general (or multiple) intelligence(s), English language proficiency, and lexile reading range competency. The assumption was that these variables would have a relatively equal effect on each of the independent variables, causing multicollinearity, and therefore should be excluded. As there is still 60% of the variance to account for, further research should include adding these as further independent variables into the model to see what the results yield. The higher unexplained variance for high school ELA could possibly be explained by the fact that the ELA PTs were more language dependent, and the student body in the project school were largely English Language Learners (ELLs).

Limitations

One limitation of the investigation was that it did not attempt to account for either the potential differences between groups, or the teachers who taught and graded each of the classes. This could be achieved in both cases by the use of hierarchical regression. By adding a categorical variable into the data set to indicate the specific teacher, it could be ascertained whether the individual teachers had a significant impact on the model. This method could also be used to determine whether there were significant differences between the teaching groups, which could point towards additional factors which account for variance. Although outside of the scope of this study, further research should incorporate these ideas.

Further limitations of the study are that only two school levels and subjects were included. Additional research should include elementary school, and STEM subjects in particular, to see

48

Habits of Mind and Performance Task Achievement

whether there are any significant differences across age-groups and academic domains. Furthermore, the study was carried out in one particular school in the SE Asian region, so to increase external validity the research needs to be broadened to include multiple schools across multiple regions.

Evaluation

At this point in the evaluation, it is useful to take a step back and consider the original and overall purpose of the study. Time is the eternal enemy of educators, and it needs to be decided which areas they should focus their time and effort upon. Is a focus on the HoMs in planning, instruction and assessment worth the effort? Upon analysis of the results, the answer would appear to be a categorical ‘yes’; however, the matter is complex and depends upon a number of factors. The HoMs themselves fall under the category of epistemic cognition, or “how people acquire, understand, justify, change, and use knowledge in formal and informal contexts”

(Greene, Sandoval and Bråten, 2016: 1). The findings appear to support a comprehensive meta-analytic review on the relationship between epistemic cognition and academic achievement (Greene, Cartiff and Duke, 2018). The authors reported “a small […] but statistically significant overall correlation between epistemic cognition and academic achievement, which could be better understood via investigations of several theoretical and methodological moderators” (p.13).

Greene, Cartiff and Duke’s work revealed four particularly relevant findings to this study. The first is that the effect sizes were higher for domain-specific measurements as opposed to domain-general measurements. This could indicate that there is an element of domain-specificity to epistemic cognition, and therefore, as within this study, it should be measured per domain rather than as a general measure. Secondly, the effect sizes were also higher when both the measures for epistemic cognition and academic achievement were aligned (domain-

49

Habits of Mind and Performance Task Achievement

general, domain-specific or topic specific). This again is supported by this study, where alignment of desired outcomes and assessments were key to the extent of correlation. Thirdly, there was a higher effect size where the academic subject within the study was aligned to the subject used in the measurement. This may indicate that just because a student can exhibit high epistemic cognitive traits in, for example, mathematics, it does not automatically follow that they can do so in say, social studies. Further study in this area is recommended by the author p.54. Finally, and of high importance, achievement instruments which measured higher order thinking, such as conceptual understandings and argumentation correlated more highly with epistemic cognition than measures of declarative or procedural knowledge. This then brings educators back to the question of what it is we value in terms of desired results. If performance assessments where students are required to transfer their learning to solve real-world problems are valued, then epistemic cognition (such as the HoMs) should be a focus of the curriculum. If we are content with students acquiring basic knowledge and skills, then they are of less importance.

It is important to note that within Greene, Cartiff and Duke's study (2018), most of the measures of epistemic cognition were actually based on epistemic beliefs as opposed to epistemic cognition. The authors themselves recognise this in their conclusions when they assert “it seems unwise to continue uncritically using self-report measures of epistemic cognition” (p. 21).

Sinatra (2016) agrees, stating “researchers must move towards defining and capturing the process of epistemic cognition in action in more nuanced ways than dichotomized belief dimensions” (p. 7). Furthermore, Sinatra espouses a move towards the measurement of epistemic practices which are defined as “how individuals use their epistemic beliefs and conceptions of knowledge in reasoning, problem solving, and decision-making” (p.12). Additional researchers (Chinn, Rinehart and Buckland, 2014) have developed the AIR model of epistemic cognition, highlighting the three components of Aims, Ideals and Reliable Epistemic

50

Habits of Mind and Performance Task Achievement

Processes. These broadly equate to goals (what students hope to achieve), beliefs (how students believe knowledge is attained), and practices (the thought processes students utilise when engaging with information). Through the development and implementation of the HoM Journal used in this study, it is the author’s assertion that these have been addressed to some extent, although further refining is desirable. One example of how the HoM Journal could be refined is to build on the work of Kelly (2016) and expand the Observable Behaviour criterion to include a more detailed analysis of the discourse and social interaction of the students throughout the course of a unit, to enable both a more accurate measurement of HoM application ,and a mechanism through which to give high quality constructive feedback at both the process and self-regulation levels (Hattie and Timperley, 2007). In summary, the research outlined in this paper supports and furthers the recent literature on the importance of epistemic cognition, and also points towards necessary areas of further research.

Chapter 5 : Conclusions and Recommendations

Conclusions

This study points towards the HoMs being a very powerful framework of cognitive traits to integrate into the curriculum. If schools are willing to acknowledge the necessity to modify the curriculum to cater for a rapidly changing world where access to almost any information necessary is at one’s fingertips, and where artificial intelligence and robots will be capable of carrying out the tasks required for a large percentage of jobs in today’s markets (Huang and Rust, 2018), then the focus of education should shift away from declarative and procedural knowledge towards a focus on the cognitive skills which are embodied in Costa and Kallick’s

(2008) HoMs. The Partnership for 21st Century Skills (2008) states “Advanced economies, innovative industries and firms, and high-growth jobs require more educated workers with the

51

Habits of Mind and Performance Task Achievement

ability to respond flexibly to complex problems, communicate effectively, manage information, work in teams and produce new knowledge” (p.6). One only has to analyse this statement briefly, and it can be seen that the HoMs Thinking Flexibly, Thinking and Communicating with Clarity and Precision, Question and Problem Posing, Applying Past Knowledge, Thinking Interpedently and Creating, Imagining & Innovating, are not only desirable, but absolutely necessary to achieve these aims.

Whilst there is little debate that this is the direction K-12 education should take, much more work needs to be done to determine what practical changes are necessary in our classrooms to make these goals a reality. Standards movements such as the Common Core State Standards (Common Core State Standards Initiative, 2018), the Next Generation Science Standards (NGSS Lead States, 2013) and C3 (The Washington State Council for the Social Studies, n.d.) have gone some way towards this, by reframing domain-specific educational goals. The Understanding by Design framework (Wiggins and McTighe, 2005) has also contributed positively by providing both a framework, and a useful set of tools and templates to enable educators to ‘unpack’ these standards into the knowledge, skills, understandings and ultimately transfer which we wish to see our students display. Furthermore, the movement towards authentic performance assessments (Center for Collaborative Education, CCE, 2017) has given us a new way to assess our students through real-word problem-solving tasks. However, what is noticeably lacking is an agreement on the epistemic practices which are required for student success, robust assessment instruments to measure these, and practical pedagogical strategies for educators in the classroom (Greene, Sandoval and Bråten, 2016).

The research contained in this paper has contributed to the movement by both supplying empirical quantitative evidence that students’ display of the HoMs correlates positively to

52

Habits of Mind and Performance Task Achievement

achievement in performance assessments, and by beginning to develop an assessment tool to

measure HoM competency. The following recommendations are made2:

Recommendations for Educators

If a school values high student achievement in authentic performance assessments, the HoMs should also be embedded in the curriculum. This recommendation is based upon the high level of correlation between HoM performance and Performance Task achievement indicated at both school levels and subjects. Figure 4.5 Middle School Social Studies Coefficients & Collinearity & Figure 4.12 High School ELA Coefficients & Collinearity

There should be enough time set aside for learning activities incorporating the HoMs, these should not be treated separately from, and should be linked to unit content. If HoM performance positively affects Performance Task achievement, it stands to reason that time should be spent in the classroom focusing on them. The study revealed that when the HoMs selected were closely aligned with transfer goals, and therefore linked to unit content, the level of correlation between them was higher. Figure 4.4 Middle School Social Studies Model Summary & Figure 4.11 High School ELA Model Summary

HoM summative assessments should be included in the curriculum in the form of a reflective journal where students are assessed on their ability to identify and display the HoMs, reflect upon their own application of them, self-assess, and set goals for improvement. This was the assessment instrument used in the study, and whilst further work needs to be done in developing and refining this tool, the findings indicated a level of validity for the instrument. This complements existing research on the positive effect

2 It should be noted that the HoMs may not be the only set of cognitive traits which yield the same positive results on performance assessment achievement, but they were the ones investigated in the study, so therefore are named in the recommendations.

53

Habits of Mind and Performance Task Achievement

metacognitive strategies have on achievement of learning outcomes. Appendix 6: Habits

of Mind Journal and Rubric

Specific HoMs should be selected for each unit of study and there should be a high level of alignment between the HoMs selected and the cognitive skills required for success in the performance assessment. This is based on the discovery during peer-review of the higher level of alignment in middle school social studies, and consequently the higher predictive power of the HoMs than in high school ELA. Appendices 7-13 p.82

Adopters of the UbD framework should include the HoMs in the Stage 1 Desired Results section, and a new section should be included in a later UbD template. The study found no collinearity between performance in HoM assessment, EQ discussion, and Knowledge and Skills attainment, so therefore this indicates that they should be treated separately in terms of ascertaining student readiness for Performance Task achievement. See pages 39 ; 45

Recommendations for Further Research

Quantitative empirical studies for the HoMs’ correlation to PT achievement should be carried out in other disciplines, particularly STEM subjects to attempt to ascertain the degree of domain-specificity.

Quantitative empirical studies for the HoMs’ correlation to PT achievement should be carried out at the elementary school level to attempt to ascertain differences between age-groups.

Quantitative empirical studies for the HoMs’ correlation to PT achievement should be carried out across a broad selection of nations, contexts and student demographics.

54

Habits of Mind and Performance Task Achievement

Knowledge acquisition and skills display should be treated separately to attempt to ascertain differences in their correlation to performance task achievement, and therefore their predictive power.

Further research needs to be carried out in developing, refining and testing HoM assessment instruments in order to maximize their validity and reliability.

Further research needs to be carried out in developing, refining and testing practical pedagogical strategies in order to determine which strategies maximize student attainment, and application of the HoMs.

55

Habits of Mind and Performance Task Achievement

Glossary of Terms

A = Acquisition of knowledge and discrete skills

AERO = American Education Reaches Out (the academic standards for social studies adopted in the project school)

CCSS = Common Core State Standards (the academic standards for English Language Arts adopted in the project school)

ELA = English Language Arts

EQ = Essential Question (overarching, open-ended questions designed to promote inquiry into big ideas and debate and discussion)

HoMs = Habits of Mind (cognitive dispositions for intelligent behaviour)

K&S = Knowledge and Skills Assessment

LTTG = Long-term Transfer Goal (synonymous with SLO in the study)

M = Meaning making of big conceptual ideas

NGSS = Next Generation Science Standards

PA = Performance assessment

PT = Performance Task

SLO = School-wide Learner Outcome

T = Transfer of knowledge, skills, understandings and HoMs in order to complete an authentic performance task

UbD = The Understanding by Design framework for curriculum planning

WASC = Western Association of Schools and Colleges

56

Habits of Mind and Performance Task Achievement

References

Accrediting Commission for Schools (2016) ACS WASC Overview | Accrediting Commission for Schools Western Association of Schools and Colleges, [online] Available at: http://www.acswasc.org/wasc/acs-wasc-overview/ (Accessed 10 August 2017).

Accrediting Commission for Schools, Western Association of Schools and Colleges (2014) Focus on Learning International Edition.

Baddeley, A. (1986) Oxford psychology series, No. 11. Working memory, New York: Clarendon Press/Oxford University Press.

Beyer, B. (1998) Improving student thinking, The Clearing House, 71(5), pp. 262–267.

Bollen, K. A. (2002) Latent Variables in Psychology and the Social Sciences, Annual Review of Psychology, 53(1), pp. 605–634.

Bouckenooghe, D., Cools, E., De Clercq, D., Vanderheyden, K. and Fatima, T. (2016) Exploring the impact of cognitive style profiles on different learning approaches: Empirical evidence for adopting a person-centered perspective, Learning and Individual Differences, 51, pp. 299–306.

Bransford, J., Brown, A. and Cocking, R. (2000) How people learn: Brain, mind, experience and school. Washington, DC: Commission on Behavioral and Social Sciences and Education, National Research Council, National Academy Press.

Campbell, J. (2006) Theorising habits of mind as a framework for learning, AARE the association for active educational researchers, [online] Available at: http://aassaquito2012.pbworks.com/w/file/fetch/51873287/Theorising%20Habits%20of%20Mind%20as %20a%20Framework%20for%20Learning.pdf (Accessed 19 December 2016).

Center for Collaborative Education, CCE (2017) Quality Performance Assessment and Task Design, [online] Available at: http://cce.org/work/instruction-assessment/quality-performance-assessment (Accessed 9 September 2017).

Charbonneau, P. C., Jackson, H. A., Kobylski, G. C., Roginski, J. W., Sulewski, C. A. and Wattenberg, F.

(2009) Developing students’‘habits of mind’ in a mathematics program, PRIMUS, 19(2), pp. 105–126.

Chinn, C. A., Rinehart, R. W. and Buckland, L. A. (2014) Epistemic cognition and evaluating information: Applying the AIR model of epistemic cognition, Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences, pp. 425–453.

Common Core State Standards Initiative (2018) Preparing America’s students for success. [online] Available at: http://www.corestandards.org/ (Accessed 26 May 2018).

Cornoldi, C., Carretti, B., Drusi, S. and Tencati, C. (2015) Improving problem solving in primary school students: The effect of a training programme focusing on metacognition and working memory, British Journal of Educational Psychology, 85(3), pp. 424–439.

57

Habits of Mind and Performance Task Achievement

Costa, A. L. (1991) Developing Minds: A Resource Book for Teaching Thinking. Revised Edition, Volume 1.

Costa, A. L. and Kallick, B. (2000) Describing 16 habits of mind.

Costa, A. L. and Kallick, B. (2008) Learning and leading with habits of mind: 16 essential characteristics for success, ASCD.

Dede, C. (2010) Technological supports for acquiring 21st century skills, International encyclopedia of education, 3.

Dewey, J. (1933) How We Think. A Restatement of the Relation of Reflective Thinking to the Educative Process, Boston etc. (DC Heath and Company) 1933.

Donovan, L., Green, T. D. and Mason, C. (2014) Examining the 21st Century Classroom: Developing an Innovation Configuration Map, Journal of Educational Computing Research, 50(2), pp. 161–178.

Edwards, J. (2014) Research on Habits of Mind, The Institute for Habits of Mind.

Ellis, R. and Humphreys, G. W. (1999) Connectionist psychology: A text with readings, Psychology Press.

Felder, R. M. and Soloman, B. A. (n.d.) Felder & Soloman: Learning Styles and Strategies, North Carolina State University, [online] Available at: http://www4.ncsu.edu/unity/lockers/users/f/felder/public/ILSdir/styles.htm (Accessed 9 March 2018).

Feuerstein, R. (1980) Instrumental enrichment: An intervention program for cognitive modifiability, Univ Park Pr.

Field, A. P. (2009) Discovering statistics using SPSS: (and sex, drugs and rock ‘n’ roll), 3rd ed, Los Angeles, SAGE Publications.

Gardner, H. (2011) Frames of mind: The theory of multiple intelligences, Basic books, [online] Available at: https://books.google.com/books?hl=en&lr=&id=2IEfFSYouKUC&oi=fnd&pg=PR7&dq=Howard+Gardner+ multiple+intelligence&ots=3-6N9LXRuW&sig=jFD6MCeMi2t74PNmuqNxGxBxMwg (Accessed 17 May 2015).

Gardner, H. (1995) Reflections on multiple intelligences: Myths and messages, Phi Delta Kappan, 77, pp. 200–200.

Goleman, D. (1996) Emotional Intelligence. Why It Can Matter More than IQ., Learning, 24(6), pp. 49–50.

Greene, J. A., Cartiff, B. M. and Duke, R. F. (2018) A meta-analytic review of the relationship between epistemic cognition and academic achievement, Journal of Educational Psychology, [online] Available at: https://search.ebscohost.com/login.aspx?direct=true&db=pdh&AN=2018-09738-001&site=eds-live.

Greene, J. A., Sandoval, W. A. and Bråten, I. (2016) Handbook of epistemic cognition, Routledge.

58

Habits of Mind and Performance Task Achievement

Hardiman, M. (2010) The Brain-Targeted Teaching Model: A Comprehensive Model for Classroom Instruction and School Reform, New Horizons for Learning, 8(1), [online] Available at: https://jhepp.library.jhu.edu/ojs/index.php/newhorizons/article/view/58 (Accessed 2 May 2016).

Hattie, J. and Timperley, H. (2007) The power of feedback, Review of educational research, 77(1), pp. 81–112.

Hawker Brownlow Education (2013) What is Understanding by Design? Author Jay McTighe explains. - YouTube, [online] Available at: https://www.youtube.com/watch?v=d8F1SnWaIfE (Accessed 24 March 2018).

Heale, R. and Twycross, A. (2015) Validity and reliability in quantitative studies, Evidence Based Nursing, 18(3), pp. 66–67.

Huang, M.-H. and Rust, R. T. (2018) Artificial Intelligence in Service, Journal of Service Research, 21(2), pp. 155–172.

Infowhelm: Global Digital Citizen (2013) InfoWhelm and Information Fluency, Vimeo, [online] Available at: https://vimeo.com/55495939 (Accessed 9 September 2017).

Jensen, E. (2005) Teaching with the brain in mind, ASCD.

Kelly, G. J. (2016) Methodological considerations for the study of epistemic cognition in practice,

Handbook of epistemic cognition, pp. 393–408.

Kereluik, K., Mishra, P., Fahnoe, C. and Terry, L. (2013) What Knowledge Is of Most Worth: Teacher Knowledge for 21st Century Learning, Journal of Digital Learning in Teacher Education (International Society for Technology in Education), 29(4), pp. 127–140.

Langer, E. J. (1989) Mindfulness. Reading, MA, US, Addison-Wesley/Addison Wesley Longman.

Lave, J. and Wenger, E. (1991) Situated learning: Legitimate peripheral participation, Cambridge university press, [online] Available at: http://books.google.com/books?hl=en&lr=&id=CAVIOrW3vYAC&oi=fnd&pg=PA11&dq=Lave+and+Weng er,+1991&ots=OBlDrpZLAm&sig=5grAhcbg7RZrmbapl6E2YRswR3Q (Accessed 5 November 2014).

Lesperance, D. J. (n.d.) Integrating Habits of Mind into the Classroom.

Liftig, I. (2009) Nurturing scientific habits of mind, Science Scope, 33(1), pp. 1–1.

Loveland, T. and Dunn, D. (2014) Teaching Engineering Habits of Mind in Technology Education,

Technology and Engineering Teacher, 73(8), pp. 13–19.

Mayer, J. D., Caruso, D. R. and Salovey, P. (1999) Emotional intelligence meets traditional standards for an intelligence, Intelligence, 27(4), pp. 267–298.

McClelland, J. L., Rumelhart, D. E. and Hinton, G. E. (1986) The appeal of parallel distributed processing,

MIT Press, Cambridge MA, pp. 3–44.

59

Habits of Mind and Performance Task Achievement

McTighe, J. (2014) Long Term Transfer Goals.

Mongan-Rallis, D. H. (2005) Six facets of understanding: Grant Wiggins, [online] Available at: https://www.d.umn.edu/~hrallis/courses/3204sp05/planning/wiggins_6facets.htm (Accessed 25 March 2018).

Nelson, T. O. and Narens, L. (1994) Why investigate metacognition, Metacognition: Knowing about knowing, pp. 1–25.

NGSS Lead States (2013) Next Generation Science Standards: For States, By States, Washington, DC: The National Academies Press.

O’Hare, L. and McGuinness, C. (2015) The validity of critical thinking tests for predicting degree performance: A longitudinal study, International Journal of Educational Research, 72, pp. 162–172.

Onu, V. C., Eskay, M., Igbo, J. N., Obiyo, N. and Agbo, O. (2012) Effect of Training in Math Metacognitive Strategy on Fractional Achievement of Nigerian Schoolchildren, US-China Education Review, pp. 316– 325.

Onwuegbuzie, A. J. (2000) Expanding the Framework of Internal and External Validity in Quantitative Research.

Ortlipp, M. (2008) Keeping and using reflective journals in the qualitative research process, The qualitative report, 13(4), pp. 695–705.

Osman, A. A. (2016) Effect of Habits of Mind Program on the Development of Positive Thinking and the Level of Learning Basic Soccer Skills Among Students of the Faculty of Physical Education, Ovidius University Annals, Series Physical Education & Sport/Science, Movement & Health, 16(2), pp. 228–235.

Partnership for 21st Century Learning (2007) Framework for 21st Century Learning.

Partnership for 21st Century Skills, A. (2008) 21st Century Skills, Education & Competitiveness.

Pashler, H., McDaniel, M., Rohrer, D. and Bjork, R. (2008) Learning Styles: Concepts and Evidence,

Psychological Science in the Public Interest, 9(3), pp. 105–119.

Pennequin, V., Sorel, O., Nanty, I. and Fontaine, R. (2010) Metacognition and low achievement in mathematics: The effect of training in the use of metacognitive skills to solve mathematical word problems, Thinking & Reasoning, 16(3), pp. 198–220.

Perkins, D. (1995) Outsmarting IQ: The emerging science of learnable intelligence, Simon and Schuster.

Pintrich, P. R. (2000) An achievement goal theory perspective on issues in motivation terminology, theory, and research, Contemporary educational psychology, 25(1), pp. 92–104.

Pintrich, P. R. and De Groot, E. V. (1990) Motivational and self-regulated learning components of classroom academic performance., Journal of educational psychology, 82(1), p. 33.

60

Habits of Mind and Performance Task Achievement

Pritchard, A. (2009) Ways of learning: learning theories and learning styles in the classroom, Abingdon, Oxon; New York, NY, Routledge.

Punch, K. F. and Oancea, A. (2014) Introduction to research methods in education, Sage.

Schneider, W. and Bjorklund, D. F. (1998) MEMORY. D. KUHN & RS SIEGLER (Eds.), Handbook of child psychology: Cognition, perception and language (pp. 467-521), New York: Wiley & Sons.

Schneider, W., Bjorklund, D. F., Valsiner, J. and Connolly, K. (2003) Memory and knowledge development, Handbook of developmental psychology, pp. 370–403.

Schraw, G. and Dennison, R. S. (1994) Assessing metacognitive awareness, Contemporary educational psychology, 19(4), pp. 460–475.

Sinatra, G. M. (2016) Thoughts on knowledge about thinking about knowledge, Handbook of epistemic cognition, pp. 479–491.

Singh, V. (2017) Exploring the Relationship between cognitive Style and learning Style with Academic Achievement of Elementary School learners, Educational Quest, 8, p. 413.

Sternberg, R. J. (1985) Beyond IQ: A triarchic theory of human intelligence, CUP Archive.

Sternberg, R. J. (2001) Epilogue: Another mysterious affair at styles, Perspectives on thinking, learning and cognitive styles, pp. 249–252.

Sullivan, P. (2012) Essential habits of mind for college readiness, College English, 74(6), pp. 547–553.

The Washington State Council for the Social Studies (n.d.) C3 Framework, Washington State Council for the Social Studies, [online] Available at: http://www.wscss.org/c3framework/ (Accessed 26 May 2018).

Tulving, E. (1985) How many memory systems are there?, American psychologist, 40(4), p. 385.

Turner, V. W. and Schechner, R. (1988) The anthropology of performance.

University of Roehampton, London (2014) Ethics Guidelines.

University of Surrey (n.d.) Unit 3 Primary and Secondary Sources, [online] Available at: http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Infor mation%20Leicester/page_21.htm (Accessed 9 September 2017).

Vollrath, D. (2016) Developing Costa and Kallick’s Habits of Mind Thinking for Students with a Learning

Disability and Special Education Teachers, [online] Available at: http://scholarworks.arcadia.edu/grad_etd/5/.

Voogt, J. and Roblin, N. P. (2012) A comparative analysis of international frameworks for 21 century competences: Implications for national curriculum policies, Journal of Curriculum Studies, 44(3), pp. 299–321.

Whimbey, A. (1975) Intelligence Can Be Taught (New York: EP Dutton).

61

Habits of Mind and Performance Task Achievement

Whiting, J. W. M. (1980) The making of psychological anthropology, Univ of California Press.

Wiggins, G. P. and McTighe, J. (2012) The Understanding by design guide to advanced concepts in creating and reviewing units, ASCD.

Wiggins, G. P. and McTighe, J. (2005) Understanding by design, ASCD.

Zakaria, E., Yazid, Z. and Ahmad, S. (2009) Exploring Matriculation Students’ Metacognitive Awareness and Achievement in a Mathematics Course, International Journal of Learning, 16(2), pp. 333–347.

Zull, J. E. (2004) The art of changing the brain, Educational Leadership, 62(1), pp. 68–72.

62

Habits of Mind and Performance Task Achievement

Appendices

Appendix 1: School-wide Learner Outcomes

63

Habits of Mind and Performance Task Achievement

Appendix 2: Approved Ethics Response Form

64

Habits of Mind and Performance Task Achievement

65

Habits of Mind and Performance Task Achievement

66

Habits of Mind and Performance Task Achievement

67

Habits of Mind and Performance Task Achievement

68

Habits of Mind and Performance Task Achievement

69

Habits of Mind and Performance Task Achievement

70

Habits of Mind and Performance Task Achievement

71

Habits of Mind and Performance Task Achievement

72

Habits of Mind and Performance Task Achievement

73

Habits of Mind and Performance Task Achievement

74

Habits of Mind and Performance Task Achievement

Appendix 3: Authorisation Letter from the Board of Directors

75

Habits of Mind and Performance Task Achievement

Appendix 4: Essential Questions Summary Sheet