Chapter 4: Findings. "A study of the relationship between Interpersonal Choices and Performance Task achievement in an International School in South-east Asia"
Author: Daryl Thompson
Chapter 4: Findings
In terms of the research question “To what extent is there a correlation between Interpersonal choices and Performance Task success?”, it was deemed that a correlational quantitative study was required using Multiple Linear Regression (MLR). The advantage of such analysis is to have the opportunity to determine what influence one or more independent variables can have on the criterion value. Does the student performance in each of the assessments have a strong correlation to the achievement in the performance task? Do any of the assessment types have no correlation or a negative correlation? Which of the assessments have the strongest correlation? Conversely, if multiple regression is not analyzed properly it can severely affect the study. Incomplete data is an example of this negative effect as well as the possibility of concluding that a correlation is a causation.
Therefore, we simply wished to use MLR to test if the cognitive disposition toward Interpersonal choices would have a positive effect on student achievement in order to justify its implementation in the curriculum.
There were 122 Grade 10 students who were enrolled for the Interpersonal unit, Phnom Penh: Opportunities and Challenges. Of these 122 students, 87 full data sets were completed wherein the students submitted each of the four assessments (Independent Variables). These were the Essential Questions (EQ), Habits of Mind (HoM), Knowledge and Skills (K&S) and the Interpersonal survey responses. The data sets were completed with the Performance Task achievement data (Dependent Variable) In terms of Stage 1 of the Understanding by Design Framework (UBD), the K&S are the declarative and procedural, the EQs the conceptual and the HoM and personalization, the cognitive. Fifteen of these data sets contained incomplete surveys, however the Interpersonal elements were ascertained by examining the choices made through the student work. Late submissions and non-submissions which scored zero formed the basis of the incomplete data sets. These data sets were omitted as they would have corrupted the analysis.
Figure 4.1 PT vs EQ Scatterplot
Figure 4.2 PT vs Knowledge & Skills Quiz Scatterplot
Figure 4.3 PT vs HoM Scatterplot
Figure 4.4 PT vs Personalization Scatterplot
In the scatter plots, we are looking for ‘cigar-shaped’ patterns which show the correlation between the independent variables and the dependent variable. This correlation is clear with the Essential Questions and Knowledge and Skills Quiz. The scores for these two assessments are grouped more in the higher end of achievement. This can be expected within the UBD framework where knowledge, skills and conceptual understandings through EQ discussions are very much embedded within the Stage 1 performance expectations. This correlation is not seen as clearly in the scatter plots for both HoM and Personalization where we see much more outliers where either students didn’t perform well in the milestone assessments but did so in the PT, and vice versa. The school has added the cognitive dispositions of the HoM more recently and the assessments have had a major shift this year to a more in the moment assessment Utilizing the Dimensions of Growth. Therefore, it was not surprising to see less of a correlation between the newly introduced cognitive elements of HoM and Personalization as it will take time to embed these into the curriculum. Indeed, six years is the normal cycle of implementation of a new initiative – ask Phil. The outliers in the Interpersonal total could also be attributable to the variants in the different Interpersonal elements so we shall look at that breakdown later. However, the model summary will show that this is less important than actually ascertaining whether there was any positive correlation or not.
Figure 4. 5 Model Summary
The R Square figure in the model summary essentially tells us how strong the model actually is. It is relatively strong as almost 40% of the variability in PT achievement can be explained by the chosen independent variables. Muscott had about 30% in his study for High School so there is some reproducibility here and indeed this model is actually slightly stronger. However, it does mean that 60.9% of the variation was due to additional independent variables, and random error.
Figure 4.6 Coefficients and Collinearity
The coefficients and collinearity table above give us specific statistical information regarding the correlation between the independent variables and the dependent variable. First of all, we can look at the p-values, the probability values, which are represented in the Sig column. These confirm if indeed there is a relationship between the IVs and the DV. Within MLR, the significance level is set as < 0.05. The p-values of all of the IVs fall below this level.
The findings indicate that the null hypothesis can be rejected. We can see with the sig or p value, the probability value of all of the independent variables are below the threshold of 0.05. The probability of the null hypothesis being true is almost zero. We have hypothesized that students’ performance in EQs, HoM, K&S and level of personalization will have a significant impact on PT achievement. The null hypothesis is that it wouldn’t. It is highly improbable that the null hypothesis is true. It is highly probable that these things didn’t happen by chance.
The next important feature of the table concerns the b-values or the beta weights. The beta weight shows you how much the criterion variable increases (in standard deviations) when the predictor variable is increased by one standard deviation — assuming other variables in the model are held constant. As we have determined that there now is a relationship, how strong is it? The Essential Questions had the highest predictive power, with a 1% increase in EQ achievement resulting in a .329% increase in PT performance. This was followed by the knowledge and skills quiz with a 1% increase in quiz achievement resulting in a .322% increase in PT performance. Then, a 1% increase in Personalization resulted in a .203% increase in PT performance. Finally, a 1% increase in HoMs resulted in a .182% increase in PT performance. Again, this has levels of reproducibility within the same school as Muscotts figure for HoM and therefore the introduction of cognitive learning to the curriculum was almost identical at .17%. It is worth pointing out however, that Mascot’s HoM assessment was a holistic journal whereas this study introduced the more in the moment HoM assessments using the Dimensions of Growth. It does somewhat validate the introduction of the DoGs which will undoubtedly become more institutionalized over the following years with expected increase in correlation with PT achievement.
Multicollinearity
In the coefficients and collinearity table (4.6), the VIF scores for all independent variables are very close to 1, which shows that multicollinearity was not an issue in the study. In other words, the variance in the IVs could not be accounted for by variance in the other independent variables. In the UBD framework there is interrelatedness between the Stage 1 desired goals as collectively they are all required in the act of transfer. For Example, in order to meet the unit transfer goal of making informed decision about how various stakeholders are affected by urban planning in Phnom Penh, students are required to take responsible risks (HoM) in order to find solutions to how individuals affect the physical environment and how are they affected by it. However, the variables are independent enough to justify measuring them in their own rights. If the independent variables influenced each other significantly then the model itself would have to be adjusted as it wouldn’t be
possible to state how much an individual independent variable influenced the dependent variable; however, this was not the case in the study.
Figure 4.7 Model Summary with Interpersonal element of feedback
Figure 4.8 Model Summary with Interpersonal element of Idea Generation
Figure 4.9 Model Summary with Interpersonal element of Task and Audience
The model summary for Task and Audience shows the strongest relationship between variables and it also has the highest predictive power of the three elements. The evidence from the student work did indeed show that the students had more success with this element than idea generation. The students seemed to have more confidence in choosing a unique audience without teacher influence in which to direct their report as opposed to actually generating a self-directed unique idea to propose solutions to a future Phnom Penh which may not have been aligned to the performance expectations and hence would have endured a potentially greater risk to high achievement in the PT. It is much more likely for a unique idea to miss the performance expectations than choosing an audience of interest with an element of teacher guidance in the idea generation.
MLR has already shown that there is a positive correlation between the combined Interpersonal elements and PT achievement. The main reason for breaking down these three elements and showing the correlation between each individual one is for the purpose of implementation and further research. The study has shown that the combination of these three elements has been successful in this unit. However, that may not always be the case. Going forward alignment within UBD needs to show the need for each element for that unit of study. It needs to show how many of the elements are required for each unit and which ones are the most effective for successful transfer.
Analysis
In the Social Sciences in general, often studies are not done that try to replicate findings. Recommendations for follow up due to limitations are common in research papers but they rarely get done. This study builds on research by Muscott within the same school and resulted in very similar findings on the correlation between the independent and dependent variables within a High School Social Studies unit. It also complements other studies where the correlation between HoM and performance achievement has been seen as well as studies where training in metacognitive strategies improved assessment performance(Osman, 2016; Vollrath, 2016). This study used the same conceptual framework as Muscott. The only addition is another independent variable, the Interpersonal learning element. This being the level to which Interpersonal learning was introduced for the students across three of the seven elements described by Kallick and Zmuda(2017). The dependent variable was the same for both studies – the performance task. The PT being the ultimate goal within the UBD framework where students transfer their knowledge skills and understandings to new and authentic situations. The findings do seem to show a lack of multicollinearity for the independent variables. It may be assumed that just because a student acquires the declarative and procedural skills through the knowledge and skills quiz that they will be successful in conceptual understandings through the Essential Questions discussion. The UBD framework is designed for the transfer of knowledge, skills and understandings into real world problem situations. Whilst the independent variables are interrelated to some extent, there is enough nuance for them to be looked at differently in terms of instruction, planning and assessment. This validates the model and UBD to an extent as that they are different enough in terms of type of learning and in terms of desired results.
With the model having a predictive power of 40%, this still leaves 60% unaccounted for. There could be many reasons for this ranging from levels of academic readiness, years within the UBD framework, English language proficiency, reading competencies and variation of teaching approaches. Future studies should include the possibility of adding such variables into the existing model to investigate what influence this would have on the results.
A Data Driven Improvement Plan (DDIP) was performed for each of the assessment types which is normal practice at the school. This is where the teachers who taught the unit and the Subject Coordinator who wrote it analyze the student achievement data for each of the assessment types. Non-engagement and Non-submission had been prevailing issues throughout the entire academic year due to the transition to online learning. However, the results were very encouraging for this unit. 81% of students submitted the Performance Task, 92% for the Habits of Mind, 84% for the Essential Questions and 86% for the Knowledge and Skills Quiz. This is much higher than the average submissions school wide and it can be certainly perceived that the students enjoyed the unit and were engaged with the extra choices and self-direction provided to them through the personalized elements. More than 75% of the students either met or exceeded the standards for each of the three criteria in the performance task which is the school threshold for intervention. Therefore, it was seen that no planning, instructional or student interventions were necessary. The two teachers who taught the unit were in their first year at the school and while it was thought they may not have fully grasped the framework and the grading system, extensive calibration was performed prior to marking with the main researcher and it was felt that the grading was sufficiently rigorous. The thresholds were also met for all of the criteria in all of the other assessment types.
Limitations
The ongoing Covid Pandemic certainly had an effect on the research. Due to government restrictions, the school was closed intermittently and there was a difficulty in ensuring continuity of practice and instruction. The study was put on hold due to this situation yet it can be said that lessons were learned in the implementation of personalization within this period. The unit began onsite but was moved to hybrid/online halfway through the eight weeks of study. Online learning may be perfect for self-directed learning but it would have been preferred to have the students onsite for longer where we could have effectively instilled the skills of self-directed learning and then let them loose in the newly constructed learning spaces in the HS campus. It was the lack of knowing how instruction was to be implemented which meant that the self-directed goals of the students were interrupted. Students who needed co-creative opportunities were not always aware of when and how they could interact with their teachers and new modes of communication were utilized which required practice to ensure profitability.
The issues with engagement due to Covid did have a negative effect on engagement and submissions which ultimately affected the sample size. As a rule of thumb, it is stated that there needs to be at least 10 cases to one independent variable, so this study did meet that criteria.
Additional limitations could include the fact that only Grade 10 students were included and only in the subject of Social Science. Additional research should include middle and elementary students and STEM subjects in particular, to gauge any differences in results across school levels and attributes across interdisciplinary areas. It would also be advantageous for further studies to extend the area of study outside the SEA region which would increase external validity.
The study shows greater correlation between the independent variables for EQs and the K&S Quiz and the PT than it does for the HoM and personalization levels. It is important to point out that the main objective was to show that there is a correlation between a range of skills and PT achievement. The UBD framework promotes a portfolio rather than a snapshot (Wiggins et al., 2005) and the study supports this by showing that a range of declarative and procedural, conceptual and cognitive dispositions are required to be successful in the summative performance task. As the predictive power of the cognitive dispositions are lower the study does point to the fact that more work needs to be done within these assessment types.
There is evidence to point towards this assumption. In Muscotts paper the HoM were assessed using a self-reflective journal. I myself conducted a qualitative action research study with students around this and it was revealed that although both saw value in the HoM they were not enamored to the journaling element. As they went through the learning activities of the unit they gave evidence statements based on their own reflections of where they believe they have employed the HoM and the benefits that brought to them. And in contrast where they felt they may have missed those opportunities and the consequences. The general feedback from both students and teachers is that these journals were not in the moment and the students struggled to remember the specifics of the identification and application of the HoM. The challenge within the school is that the value of cognitive dispositions is not in question but the best way to assess them has been. Some elements within the school even question the efficacy of assessing them at all. This brings into question the purpose of assessment in general. It cannot be seen as simply a set of figures to produce for parents but used as data for feedback and further tailoring of instruction for students to acquire the dispositions. Therefore, work carried on to improve this area. This experimentation has certainly affected the correlation with the final performance task. In this study, the HoM assessment was conducted using the Dimensions of Growth (DoG) for the first time. This moved the assessment away from the afterthought of the journals to more in the moment assessments which are much more tied to content and not to be seen by students and teachers as something separate – it’s not an add on. There is an evolution of thinking, if we embed activities throughout the unit, giving more practice on a very specific version of the habit within the context of the unit, we hypothesize that the predictive power is going to go up. We had two habits for this unit – creating and innovating and taking responsible risks which fit into the context. The HoM is so rich and deep that it is really only six years after their implementation that we are now really thinking about the five dimensions. These dimensions are meaning, capacity, alertness, value and commitment. What we later realized and this was before our understanding of the dimensions was as deep as it is now was that actually the journals only considered some of the dimensions. There was a need to embed rather than to add on. We were actually focusing on value and commitment within the journals but that wasn’t quite enough. It’s not to say that we were not employing the other three because they are so intertwined but at this point it was not as intentional. The dimensions were how to take it to the next level. The feedback from both students and teachers in terms of the dimensions of growth assessments as opposed to the more traditional generic journals has been very positive and it is hoped that further work on these will improve the correlation between HoM assessments and PT achievement.
icons on the left to see existing comments.
0 General Document comments
0 Sentence and Paragraph comments
0 Image and Video comments
General Document Comments 0
