Chapter 3 of "Exploring the Relationship Between Social-Emotional Competencies and Student Outcomes in Online Learning Environments"
Author: Sarah K. Teeple
Chapter Three
Introduction
The lack of rigorous research regarding online programming for K-12 students influenced the decision to employ a mixed-method, quasi-experimental design in the present study. This study sought to clarify the existence of any positive correlations between student social-emotional dispositions and the dependent variables of social presence and student performance. In addition, open-ended questions and follow-up interviews helped to further extend the narrative about what qualities or characteristics of online pedagogy are most positively perceived by students, and which student habits and attitudes are most facilitative of a positive online learning experience. The purpose of the study was to provide evidence and support for specific types of teacher training and practices for the benefit of online learners.
Role of the Researcher
The researcher is a doctoral student at a private university in Montgomery County, Pennsylvania who was employed as a second grade general education teacher within the district of study. She worked previously as a fourth grade teacher in the district of study, and taught first grade in a dual-language Spanish/ESL program in Denton County, Texas. She has prior experience serving in the educational community as a licensed professional counselor in the state of Texas.
Due to the proximity of the researcher to the participants in the study and the participants’ teachers, there was a risk that the investigator could experience conflicts of interest between her professional role as an educator and in her role as a researcher. These risks were mitigated by having the researcher maintain a log of all communications received or initiated by the researcher regarding the nature, content, or purpose of the study. Recruitment communications were shared with the superintendent’s office and building principals. Further, the investigator was not a teacher in the grade level of students being recruited for this study, nor was the investigator employed in any supervisory role with possible teacher participants. Additionally, the researcher did not personally know or directly work with any of the teacher participants before the study took place. Care was taken to ensure all participants received verbal and written notification that the purpose of the study was not evaluative of teachers or students, nor was it intended for the purpose of directly modifying the district’s current virtual learning structure or offerings. The researcher applied reflexivity to the recruitment and analysis processes by debriefing with peer researchers, and by identifying and disclosing possible conflicts of interest prior to implementing the recruitment and data collection phases of the study. Possible biases and conflicts of interest were also mitigated by preserving anonymity throughout the data analysis and transcribing the narrative responses without identifying student or teacher names.
Study Design
The lack of rigorous research on the outcomes of K-12 online learning is a theme reiterated by the U.S. Department of Education (2010), and highlights continued difficulties and limitations in drawing conclusions about the efficacy of K-12 online programs. Therefore, a mixed-method study on the extent to which social and emotional competencies correlate with online social and cognitive presences and student outcomes could provide more concrete evidence in support of the use of certain pedagogical strategies and behaviors to improve online learning experiences. The research questions are as follows:
- To what extent do social and emotional competencies predict online social presence and student performance in fully remote instruction?
- How do social and emotional competencies manifest in fully online learning environments?
- How are adaptive student behaviors and attitudes reinforced and modeled through remote instruction?
A cross-sectional study in which a baseline for social and emotional dispositions determined the groupings of the participants was the basis for this design. Likert-scale assessments using revised versions of established instruments were administered. Based on the results of this assessment, students were sorted into groups. Students who intuitively seemed to demonstrate the requisite social-emotional (SEL) habits comprised Group 1. Students who lacked an initial predisposition towards the studied SEL habits constituted Group 2. The reliability of the data was bolstered by triangulating student self-assessments with teacher and parent assessments. The students then completed surveys to assess for perceptions of online social presence, and the groups were compared. In addition, summative grades were gathered. The researcher hypothesized that student dispositions would be positively related to student perceptions of social presence in online learning environments (OLEs) and that when students show evidence of certain Habits of Mind, they would also show evidence of higher performance and more positive perceptions of their learning engagement in OLEs. Conducting follow-up interviews also helped to clarify the overall picture of student experiences in online learning from multiple perspectives.
Population and Sample
This study was conducted in a suburban school district approximately 30 miles north of Philadelphia, Pennsylvania, where the researcher is employed as a classroom teacher. This district was chosen due to its ongoing work with Dr. Kallick to incorporate Habits of Mind into the district curriculum. Participants were recruited from across the seven district elementary schools, serving approximately 1,475 students in the intermediate grades three through five, through email requests and electronic communications. Nearly 20% of students in the district qualified for free and reduced lunch, and the student population was predominantly (approximately 85%) white. About 7% of the student population identified as Hispanic or Latino, 2.9% as multiracial, 2.6% as Asian, and 2.2% as African American. Roughly 20% of the district’s intermediate students were participating in fully remote online learning at the time of the study, and therefore the population consisted of approximately 120 virtual students in fifth grade. The researcher conducted convenience sampling to recruit students from the population of general education students participating in a fully remote writing class. From the sample set, additional student participants and virtual teachers were selected for individual interviews.
Data Collection and Instruments
A Habits of Mind (HoM) survey was administered to a sample of students and their parents and teachers, and mean triangulated scores demonstrated baseline levels of student social and emotional dispositions. The Social Presence Dimension of the Community of Inquiry Questionnaire was administered at a later date to the sample students only, and variances in both student perceptions of social presence and summative writing evaluations were evaluated with respect to initial HoM scores.
Habits of Mind survey data was collected via student self-report. To help ensure reliability of the student ratings, student Habits were assessed by a parent or legal guardian, and the current teacher of record. The multiple ratings were triangulated, creating a mean Habits of Mind score for each student cumulatively across the six social-emotional dimensions being studied. Students with fewer than three separate reports were excluded from the study. The grouping factor for analysis was the extent to which students appeared to be predisposed towards certain Habits of Mind that were deemed most relevant for writing courses by the researcher. Out of the 16 original Habits of Mind, the researcher chose to focus on assessing student dispositions across the following six categories, as shown in Table 3.1: Persisting (Habit #1); Thinking Flexibly (Habit #4); Thinking About Your Thinking/ Metacognition (Habit #5); Striving for Accuracy (Habit #6); Thinking and Communicating with Clarity and Precision (Habit #9); Creating, Imagining, and Innovating (Habit #11). The six Habits were chosen to reflect the scope of the research project. The Habits were selected based on the researcher’s perceived relevance to the subject matter being studied and based on Muscott’s (2018) recommendations to bolster construct validity by including overlapping or similar dispositions.
Table 3.1
The Habits of Mind rating scale that was modified for the present study is an ordinal scale for describing proficiency in each of the six social and emotional predispositions of interest, with three sub-questions per Habit. Ratings of three or four on each sub-question represent proficient or exemplary student habits, respectively. Students who averaged a score of three or higher on each of the 18 questions (or who had a cumulative score of 54 or higher out of a maximum score of 72 points) represented the intuitive group (Group 1), and students who averaged fewer than three points per question (or fewer than 54 points cumulatively) represented the non-intuitive group (Group 2).
Perceptions of online social presence were self-reported by students at the conclusion of the unit of study using the Social Presence Dimension of the Community of Inquiry Questionnaire, and were scored on a 5-point Likert scale for each of nine questions. The Cognitive Presence and Teaching Presence dimensions of the full Community of Inquiry Questionnaire were omitted because their relevance lay outside of the scope of this research study. The Social Presence dimension of the Community of Inquiry Questionnaire was modified for accessibility to elementary students in terms of readability and comprehension. The number of items in the Social Presence dimension were preserved, but substitutions or omissions in wording were chosen in order to reflect the age and experience of the participants. Cumulative scores for social presence were calculated in the spreadsheet for each student, between a minimum of nine points and a maximum of 45 points.
Student performance was calculated using a unit published writing scoring rubric. Students were assessed by their teacher on a final piece of published writing. A writing piece received a numerical score for each of nine categories, with a minimum value per category of “one” and a maximum value of “four”. Cumulative writing scores per student ranged from nine to 36.
From the sample set, an additional seven student participants were selected for follow-up individual interviews. In each interview, students and their guardians received written and oral descriptions of the six Habits of Mind being studied. Students and guardians were asked four questions regarding how they perceived these Habits to impact student learning and engagement: 1) In what ways do you see these habits helping you in your learning in an online class? 2) In what ways do you see these habits helping you be engaged in online learning? 3) When you’re faced with a challenging task, which of these habits do you find are most helpful? 4) In what ways does your teacher help support the application of these Habits in your learning? Three fully online teachers were also selected for semi-structured individual interviews. They were prompted to reflect on the ways in which they saw the Habits of Mind influencing student learning and engagement in fully online classes, and the ways in which their pedagogical practice could better support students in developing and applying these Habits: 1) In what ways do you see these habits helping students in their learning in an online classroom? 2) In what ways do you see these habits helping students be engaged in online learning? 3) When students are faced with a challenging task, which of these habits do you think are most helpful? 4) In what ways does your teaching practice help support the application of these Habits in student learning?
All data was collected through confidential, password-protected Google forms, in an account to which only the researcher had access. Data was automatically populated in Google sheets. Student and guardian names were collected to allow for follow-up communications for additional phases within the study, but were cleaned from the aggregate data files used for analysis.
Data Analysis
All data was aggregated by student: Habits of Mind self-score, Habits of Mind parent score, Habits of Mind teacher score, Habits of Mind average score, grouping factor, social presence score, and student task performance score. Any missing data sets or data sets that were found to lie outside of the expected range of scores resulted in all of that student’s data being excluded from the study.
Before running descriptive statistics and analysis on the data, identifying student information was removed from the data set to provide more protection against breaches of confidentiality with regards to student information.
The cleaned data set underwent a Multiple Analysis of Variance (MANOVA) in order to identify the relationships between social-emotional groupings and both dependent variables: student task performance and online social presence. The MANOVA was run first to test for significant associations between the independent variable groupings and both dependent variables, and was chosen to help minimize the likelihood of experiencing Type I errors, or incorrectly rejecting the null hypotheses. Effect sizes, power, and homogeneity were described as a result of the MANOVA. If significant relationships were reported via the MANOVA, additional, one-way ANOVAS and t-tests were run to identify mean differences between each group and further operationalize the strength of the relationship between Habits of Mind and each dependent variable separately.
Interviews were transcribed by the researcher, who then hand-coded the responses based on emerging themes and categories. Thematic content analysis focused on student behaviors and attitudes that support content-area skill development. In other words, the researcher was seeking to explain the ways in which student behaviors and attitudes reinforced or modeled throughout instruction.
Reliability and Validity
Arbaugh et al. (2008) report evidence of construct and external validity of the Community of Inquiry Questionnaire following a Principal Components Analysis of data collected from a multi-institutional sample of 287 participants. The results of their study demonstrate support for the ability to use the multiple dimensions of the Community of Inquiry framework as a valid and reliable predictor of student outcomes. Specifically, internal consistency of the items within the three dimensions of the Community of Inquiry instrument were supported by Cronbach’s Alphas of 0.91, 0.95, and 0.94 for the dimensions of Social Presence, Cognitive Presence, and Teaching Presence, respectively. Construct validity was evidenced by 61.3% of the total variance being attributed to the three factors, or dimensions, of study. However, it is important to note that the Community of Inquiry instrument was validated using samples of students in graduate-level classes at different institutions of higher learning in both Canada and the United States. The existing survey, with permission from the original authors of the survey tool and under the Creative Commons license as an open resource, has been adapted to fit the comprehension and reading level of fifth grade students. In addition, 15 out of the original 24 items on the survey were omitted, so that students only responded to the nine questions that pertain directly to the social presence dimension. This has the potential to weaken the validity of the measure used in the present study, although the researcher sought to retain the essence of each original question regarding perceptions of social presence.
Habits of Mind as a quantitative rating scale has a smaller body of research to support its validity, particularly with regards to the homogeneity and construct validity of Habits of Mind checklists. As Muscott (2018) explains, construct validity in the assessment of Habits of Mind is threatened by the possibility that some items can, in fact, be measuring overlapping or indistinct constructs. In the present study, the researcher sought to mitigate these threats to construct validity by including Habits that do not appear to be mutually exclusive, and that are relevant skills for the subject area within which the Habits are studied. For example, Muscott (2018) asserts that metacognition is a prerequisite to any social and emotional disposition for effective thinking and learning, and therefore is an essential construct to include, regardless of the performance task or specific content area of interest. Further, concerns over homogeneity within each of the 16 constructs led to the decision to include both Striving for Accuracy (Habit #6) and Thinking and Communicating with Clarity and Precision (Habit #9) in the shorter, adapted version of the Habits of Mind Likert-scale survey used herein. Similarly, both Thinking Flexibly (Habit #4) and Creating, Imagining, and Innovating (Habit #11) were also included.
External validity for HoM ratings is also limited, but studies like Muscott’s (2018) are adding to the growing body of research in support of the theory that Habits of Mind can be predictors of student performance achievement. However, future studies are needed to provide more support for the generalizability of conclusions drawn about K-12 student performance as it relates to student dispositions.
In an effort to bolster both the reliability and validity of the adapted checklist used in the current study, the researcher triangulated ratings provided by the student participants themselves with ratings provided by the teacher of record and one legal guardian for each student. The mean cumulative Habits of Mind score of the three separate ratings was the value used to represent student proficiency in the application of the six Habits of focus. Lastly, the qualitative, explanatory focus of the open-ended questions was to provide further evidence of the validity of Habits of Mind as a predictor of social presence and student task performance, as well as the validity of using social presence as both an outcome of social-emotional dispositions and as a predictor of student achievement.
icons on the left to see existing comments.
0 General Document comments
0 Sentence and Paragraph comments
0 Image and Video comments
There are a couple of changes I would like to make though:
1) Replace ‘frequency’ with ‘to what extent’
2) Frame the questions with regards the problem that the students are faced with (see the work that James Anderson is doing with the student and problem profiles)
New Conversation
Hide Full Comment
General Document Comments 0
