Chapter 3: Research Methodology and Design. "A study of the relationship between Interpersonal Choices and Performance Task achievement in an International School in South-east Asia"
Author: Daryl Thompson
1 changes, most recent almost 3 years ago
Chapter 3: Research Methodology and Design
The literature review persuaded the school within the study to implement an interpretation of PL which allows for greater student control within the classroom and within the assessments themselves. This chapter will look at how the school has adopted this version of PL and what it looks like within the UBD framework. The chapter also aims to provide the necessary background required for the Multiple Linear Regression (MLR) analysis that will follow in Chapter 4. MLR tests the correlation between the independent variables (IV) and the dependent variable (DV) and the strength of that correlation. The IVs constitute the different categories of the desired results within Stage 1 of the UBD framework. The declarative, procedural, conceptual and cognitive are all assessed in order to prepare feedback for the students on their journey to the PT which is the dependent variable and the ultimate transfer of these learner outcomes. The independent alignment between the IVs and the DV will be shown within the following sections. This will show that everything possible has been done in order to set up the MLR for success and to ultimately answer the Research Question “To what extent is there a correlation between Interpersonal choices and Performance Task success?”
Whilst there is a spattering of qualitative research which shows an apparent correlation between student choices and achievement in authentic problem solving performance tasks, there are none which have looked at a strictly quantitative relationship.
This research looks at Interpersonal choices made by students as a goal for success in the performance task, and indeed life. Whilst this is the goal, it is obvious that at such an early point of implementation, the students will be at various stages within the PL spectrum. The study did not expect that there would be a significantly high number of students electing for complete student autonomy within each of the Interpersonal elements only that where they did move away from teacher generated instruction that there would be an increase in achievement. The results of this correlation would either then lead to the endorsement or rejection of the school to further implement PL within the school.
Conceptual Framework
When Lave and Wenger (1991), espoused the idea that learning should be aligned with the social contexts within which it is useful and relevant, they initiated a fundamental transformation in terms of how learning happens. Essentially they were describing authenticity, a premise with which the school used in the study values as an essential tenet for 21st Century learning and success.
The school within the study has been using the conceptual framework of Understanding by Design (UBD) for around ten years. UBD is a curriculum planning framework which focuses on developing a curriculum and learning experiences that will help students develop and deepen their understanding of important ideas so that ultimately they can transfer their learning. This goes well beyond the memorize and recall curriculums still prevalent today. Problem solving runs right through the curriculum of this school and forms part of all the desired results. For example, one of the 12 Schoolwide learner Outcomes (Appendix 3 – List of SLOs) is Identify existing and emerging problems and formulate, justify and apply effective and optimized solutions. The HoM are cognitive dispositions used with problem solving as its goal. The school wishes to test personalization of learning as it believes that this will allow students to be engaged in rich dialogue, approach a complex problem, and sketch an idea.
Wiggins et al. (2005), stress the importance of compiling evidence of student understanding. This is achieved through a portfolio not a snapshot. The combined and varied desired results of a unit of study can be assessed in different ways to show this understanding. These desired results are seen as the first of three stages in the UBD backwards design framework. The learning objectives of any UBD unit will incorporate knowledge, skills, understandings and standards. The school has also included the SLOs into Stage 1. They have been carefully crafted to include any possible problem solving scenario and are valueless. This is an absolute must for the successful implementation of personalization into the curriculum. It is imperative that there are no restrictions when students consider choices and interests for their mode of study. Although this may seem as a logical step it is observed that this school is very much in the minority as most are influenced by religious, political, state or institutional constraints. Without these constraints, the school can successfully engage in issues which some would shy away from. Currently the school has units on such diverse issues as Immigration, Mental Health, Justice, Gender, Eurocentric influence on Asia, Cryptocurrency, Questioning the Status Quo and the ongoing Covid Pandemic and how governments have been handling it.
The school in question has added the cognitive dispositions of the HoM to its stage 1. This is not part of the UBD template but the school as a research based community is always looking at new initiatives to enrich the UBD experience. Educators such as Kallick and Zmuda (2017), have also looked at the link between UBD and the HoM and more recently the connection between the HoM and personalization. This study is also looking to ascertain the feasibility of additionally adding Interpersonal elements to the UBD framework and Stage 1.
Alignment between the three stages is vital which is why analytic rubrics linked to these desired results are shown in every assessment. One of the most important keys to first-class curriculum design is complete alignment between identified desired results, evidence collected through rigorous and diverse assessments, and rich learning activities which facilitate learning, practice and high-quality and timely feedback (Muscott, 2018). This is also valuable for focused feedback and data tracking. The alignment is clearly shown in each assessment in the criteria section of the GRASPS. This is an important tool in the UBD armory where problem based assessments are framed within a format which includes Goal, Role, Audience, Situation, Product and Standards for success. These assessments form Stage 2 of the framework, whilst Stage 3 incorporates the instructional path which is where you will find the resources and suggested lesson outlines.
Research Methodology
Much research has been constructed along the qualitative, quantitative divide with strong advocates on either side. However, Newman et al. (1998), view this as a false dichotomy and believe that all behavioral research incorporates elements of both constructs. Newman sees it as a continuum, what are known as qualitative methods are frequently beginning points, foundational strategies, which often are followed by quantitative methodologies. In this study, the decisions on the focus on cognitive dispositions very much came about as a result of qualitative Action Research within the school where there was a positive perception from both teachers and students to the benefits of cognitive learning and the success of the HoM to prepare students for their PT. (Reference Normans). This focus was then used to pose a hypothesis on the possible influence of these dispositions on the curriculum and student learning. The proving or disproving of hypotheses is then done with a quantitative approach.
Multiple Linear Regression (MLR) was used to estimate the relationship between the various independent variables (IVs) and one independent variable (DV). The regression can determine whether there is a relationship between the variables and measure the amount of that variation for each individual IV. The UBD framework combines the learning outcomes in Stage 1. These outcomes were categorized as IVs in the form of Knowledge and Skills, HoM and Essential Questions. These goals formed three of the four IVs. The fourth, Interpersonal choices were not assessed alone, but the correlation as an IV was obtained through scores given through surveys.
The specific elements of MLR required to analyze the correlation between IVs and the DV will be discussed in much greater detail within the findings in Chapter 4.
Research Design
Within the school, it was perceived that the Social Studies curriculum was very strong with its alignment to the purpose and principles of UBD and constructivist learning theories in general. Therefore, the participants of the study were to be derived from this subject area. There was some concern over engagement and submission of assessments due to Covid, therefore there were considerations over which students and teachers to select. After some thought, it was decided to select the Grade 10 students who were seen as a very driven set of students with two young, enthusiastic teachers who were very interested in testing new approaches to pedagogy. The Grade 10 task was also seen as the most conducive to introducing elements of choice within the already established goals and structure of the Performance Task,.
In this chapter we will look in more detail at each of the independent variables, the declarative, procedural. conceptual and cognitive and how they correlate to success in the summative performance task of the unit. This will allow us to compare the independent variables and ascertain whether cognitive learning in general and Interpersonal choices in that learning are viable pedagogical options for the school curriculum in its quest to produce problem solvers equipped with the skills for success in the 21st Century.
This chapter will specifically look at the constructivist school and how cognitive learning is implemented in such schools and why is perceived that this approach is more conducive to any traditional or neoliberal approaches in relation to providing students with the opportunity for personalization which suits their needs and interests.
Performance Task – Transfer
The unit of study chosen for testing Interpersonal choices was a Grade 10 Social Studies unit entitled – Phnom Penh – Opportunities and Challenges. The students had a choice to focus their research within either human, physical, economic, social or cultural issues with a view of improving quality of life in cities and towns around the world. The unit was chosen for its authenticity and relation to the transfer of problem solving skills, conceptual and cognitive ideas. It was also deemed as a task which could lead to choices within the goal, role, audience and situation within the GRASPS template.
Since introducing UBD, the school has incorporated the PT as the task which allows for the transfer of knowledge, skills and understandings (Appendices 4-5 PT and guide). Recently, it has endorsed cognitive learning theories which has meant the inclusion of the HoM into this process. More recently, choices have been made available to students within the GRASPS format which has led to the suggestion that a more formalized Interpersonal learning approach may be suitable as a further addition.
Before attempting the PT, the learner needs the opportunity to practice the declarative, procedural, conceptual and cognitive dispositions deemed necessary for success, receive feedback, and also self-assess and set goals for improvement.
Knowledge and Skills – Declarative and Procedural
For the unit of study of this research the knowledge and skills quiz (Appendix 6 – Knowledge and Skills Assignment) was framed with GRASPS and with a problem solving challenge. The students were tasked to choose three cities (not Phnom Penh) and to assess their desirability. They practiced the skills that would be required in the PT such as identifying and describing the relationship between people, places, and the environment and recognizing relationships among patterns and processes.
Essential Questions – Conceptual
The school has included an original set of open ended essential questions which encourage debate and discussion which is the catalyst for the first blossom of student opinion and perspective. In this unit those EQs were, “Is globalization Positive” and “How do we affect the physical environment and how are we affected by it?” . These questions were coupled with desired understandings to determine what inferences they should be making toward the big ideas of the unit (Appendix 7 – EQ Discussion Rubric). Student understanding of these questions would enable them to consider the consequences of globalization on Phnom Penh in order for them to provide solutions for its future. They would also consider the environmental impact and its effect on human societies within their solutions.
Habits of Mind – Cognitive
The HoM nurture problem solving, and Interpersonal learning gives it a practical framework. This culminates in the production of students who are accustomed to the ever changing dynamics of the modern world. The HoM form an integral part of the unit PT with creating, imagining and innovating being a requirement for idea generation. The employment of the HoM within the lessons also gives the student context to help with the decisions made in the idea generation Interpersonal survey. In fact, the actual HoM assessment (Appendix 8 – Habit of Mind Assessment) for the unit served this purpose as well. This was to Take Responsible Risks in the context of a smart city initiative in Saudi Arabia. The students were asked to consider why it is important to take risks when planning urban areas. This was an in the moment task Utilizing the Dimensions of Growth (DoG – Appendix 9) – value and meaning. The use of the HoM proved to be an important asset in the Interpersonal choices the students were making throughout the unit.
This was the first academic year in which the school introduced the DoG to HoM assessments. These were deemed to be much more focused than the previous journals and are articulated in a way that makes it explicit to both students and teachers what these HoM actually ‘look like’ in the context of a particular unit . It was only when implementing the DoG that it was realized that the journals which used to form the assessment actually were actually focusing on two of them – value and commitment. Focus was then added to the other three – meaning, value and commitment. The DoGs also eliminate redundancy as the students are sometimes more accustomed to what the Habits are but they need to know what those habits look like in that particular unit of study.
The HoM assessment helped the students to be more creative in the solutions they provided in the performance task and for them to be confident in taking responsible risks through their Interpersonal choices in the task.
Personalization – Cognitive
Interpersonal learning as a concept may have been new to both students and teachers but in fact choices had increasingly been made to the PTs throughout the school, it just had not been formalized as a concept. As there is not a unique assessment of Interpersonal learning as with the other IVs it was thought best to introduce these elements into the PT assessment itself. However, in order to gauge correlation between personalization and PT achievement there was a need to come up with a ‘Interpersonal score’. This was achieved through surveys (Appendices 11-13) wherein students were asked to look at the descriptors for the three levels of personalization for each element. Through negotiation with their teacher they then settled on either teacher generated, co-created or student generated. This was then numerized by adding a score of 1 for teacher generated, 2 for co-created and 3 for student generated. This was then translated to a percentage to fall in line with the data from the other IVs.
The Interpersonal choices were introduced to the unit using three of the seven elements which Kallick and Zmuda (2017) had developed. It was decided to introduce three of those elements which were most aligned to the GRASPS PT assessment documents which the students have been using for many years. These elements were task and audience, idea generation and feedback.
Paragraph 34 changes
The beauty of research is that it is never linear.
There is always a need for self-reflection and reassessment of approach of methodology.
Student responses can be unpredictable and of course it is important to keep the students informed of the details and purpose of the research.
It was initially observed that many students automatically chose teacher generated for all three elements.
This may have been due to the students feeling slightly reluctant to jeopardize the scores on their assessment which is understandable as those scores form part of their GPA for university admittance.
It was also observed that higher performing students requested teacher generated feedback, and that the lower, requested student generated feedback.
This may not be as counter intuitive as it sounds as perhaps it was conceived that the feedback from teachers may not be the kind of feedback they needed!
Teachers may give feedback on conceptual understandings but maybe the students were struggling to read a map and needed some basic skills feedback.
Instead of talking about readiness it was just being more self-directed.
This work helps in creating student profiles to help with choices as well as tailoring instruction to specific needs and requirements (Anderson, 2019).
Of course, there is nothing wrong with a student wanting to talk to a teacher, it just depends where you are on the Interpersonal continuum and slider.
The spectrum is not hierarchical, the surveys simply are trying to work out if the degree of self-directedness contributes to achievement.
That choice in itself is self-directed and metacognitive.
It was at this point it became apparent that negotiations should form part of this selection. The positives and negatives of self-directed learning were discussed within the perception that it may indeed encourage creativity and focus the task in areas of student interest which may indeed result in even better scores than if the teacher actually had more of an input. The students were made aware of the consequences and potential benefits of making their own choices which may in fact increase their achievement in the PT. Indeed, a series of cornerstone lessons were enacted to help the students practice the consequences of their choices to better inform their choices.
Triangulation formed the basis of the selection between teacher and student using prior performance assessments, dialogue, conference, observation from the classroom teacher and student self-reflection.
Personalization relates to the transfer in the performance task as the students decide on the choices best suited for them in order to achieve success. The students are progressively moving through the stages of personalization through voice, co-creation, self-discovery and social construction – this is where the transfer is.
So we have shown that the UBD framework aligns each of the IVs to the DV in order to give students the greatest chance for success in the PT. In Chapter 4 we will test the strength of each correlation to see if PL is worth implementing and to gain some insight on what that implementation might look like.
Validity and Reliability
For a research project to be successful, it must be both reliable and valid. Numerous steps within the study were taken for this to be apparent. A research journal documented reflections, questions and decisions regarding problems associated with the data collection. It functioned as a “running record” of interactions with the data (Merriam, 1998, p. 223). This negates any question of positionality or bias. As a member of the management team, it may be assumed that initiatives which are favored will simply be sanctioned. This is not conducive to the framework and philosophy that the school operates in and as a center of research, empirical evidence is also required before implementation of projects. The fact is that the researcher had little idea about how Interpersonal learning may benefit student learning and achievement within the school’s framework. It needed to be tested.
Calibration Sessions were held for each independent variable where student work was scored against the rubric by the teachers and the main researcher with the aim of forming a consensus. This ensured that the teachers who were grading the students throughout all of the assessment tasks were interpreting the performance expectations within the rubric in the same manner.
Validity can be described as “the extent to which a concept is accurately measured in a quantitative study” (Heale and Twycross, 2015, p. 66) or in other words, whether or not we are truly measuring what it is that we claim we are. There were a few concerns over this during the course of the study. The Interpersonal element of feedback was only being administered at the task, audience and idea generation levels through the surveys. The surveys did not ask students to choose how general feedback was to be administered. There were some concerns that this may not show a true correlation between feedback and PT achievement. However, the teachers involved stressed that feedback is generally consistently and effectively delivered at the task, process and self-regulation levels as it forms part of their evaluative process and is institutionalized within the school. Therefore, the validity of this correlation was deemed effective.
There was also a concern over the validity of the correlation between the HoM assessment and the PT. There was only one assessment given and this was at the very start of the ten week unit. Was this enough to have formed a valid variable? There was a question over whether the students remembered the problem solving skills they had practiced in this task when they were transferring their understanding during the PT. However, the HoM are so embedded within the curriculum and are required to be aligned to every learning activity it was deemed that even though they were assessed early they were practiced and reinforced throughout the unit and were fresh in the minds of the students during the summative task.
Ethical Considerations
The data needed and collected during the course of the research was deemed to be secondary. This is data which has already been collected in the past and was not required exclusively for the study in question. Student achievement data within the school is used for reporting purposes and is stored within Schoolbase, the school’s information system. The personal data of the students was not required as there was no analysis of student demographics required. As all of the student achievement data was anonymize in the findings, there was no need to obtain consent from students or parents. The consent of the school owners to perform the research was obtained via a signed official letter (Appendix 10: Authorization Letter from the Board of Directors).
The topic of personalization was deemed not to contain any political, religious or moral considerations which may not be openly discussed within the context of my school, community or country of residence. The investigation, findings and recommendations were not considered harmful to participants or their families and indeed the only intention was to improve the mode of study within the school.
Issues of positionality or reflexivity were negated through the use of a reflective research journal and the involvement of close colleagues as ‘critical friends’. This was vital in the effort to eradicate any bias as the main researcher had been an advocate of PL within various meetings of the International Habits of Mind research community. Throughout the research process this transparency was required but there was a need for the critical friends to analyze and criticize the literature and data in order to ascertain whether PL actually was something that was in the best interests of the school and its students.
icons on the left to see existing comments.
0 General Document comments
0 Sentence and Paragraph comments
0 Image and Video comments
The findings of the paper showed a positive correlation between achievement in both the HoM and increased personalised choices with achievement in the summative performance task. This gave encouragement to look further at HoM integration into the curriculum as well as introducing personalised choices to the performance tasks. The question arises as how to best do this.
For personalised choices, three of the seven elements were introduced – feedback, task and audience and idea generation. The latter two had direct alignment to the GRASP and can be evidenced within that document. these documents amongst others are placed within my kumospace
New Conversation
Hide Full Comment
The quest for making stage 1 goals more relevant to the unit has incorporated work on topical essential questions as well as translating the schoolwide learner goals to unit transfer goals
New Conversation
Hide Full Comment
This shows that when designing a unit the choice of which HoM to focus on and assess is an important factor in the interrelationship between stage 1 goals.
New Conversation
Hide Full Comment
Are surveys the best way to determine readiness for personalisation?
New Conversation
Hide Full Comment
What data points are necessary for triangulation of data to determine readiness for personalisation?
Is it automatic that the more self-directed your choices are the greater success you will have in the performance task?
New Conversation
Hide Full Comment
What should be included in such a conversation?
New Conversation
Hide Full Comment
Any other data points?
What are the next steps for implementation? How do we integrate the elements that have not been considered within this study and are not directly aligned to the GRASP?
The considerations for design document which has been posted in kumospace has stared to look at this issue.
New Conversation
Hide Full Comment
General Document Comments 0
