Active learning increases student performance in science, engineering, and mathematics

 

Scott Freeman a,1 , Sarah L. Eddy a , Miles McDonough a , Michelle K. Smith b , Nnadozie Okoroafor a , Hannah Jordt a , and Mary Pat Wenderoth a

 

a Department of Biology, University of Washington, Seattle, WA 98195; and b School of Biology and Ecology, University of Maine, Orono, ME 04469 Edited* by Bruce Alberts, University of California, San Francisco, CA, and approved April 15, 2014 (received for review October 8, 2013)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

4 Paragraph 4 changes Downloaded from https://www.pnas.org by 129.19.63.117 on March 21, 2024 from IP address 129.19.63.117. 0 0

The results raise questions about the continued use of traditional lecturing as a control in research studies, and support active learning as the preferred, empirically validated teaching practice in regular classrooms.

 

constructivism | undergraduate education | evidence-based teaching | scientific teaching

 

l ecturing has been the predominant mode of instruction since universities were founded in Western Europe over 900 y ago

 

Although theories of learning that emphasize the need for students to construct their own understanding have challenged the theoretical underpinnings of the traditional, instructor-focused, “ teaching by telling ” In the STEM classroom, should we ask or should we tell?

 

The answer could also be part of a solution to the “ pipeline problem ” that some countries are experiencing in STEM education: For example, the observation that less than 40% of US students who enter university with an interest in STEM, and just 20% of STEM-interested underrepresented minority students, finish with a STEM degree (5).

 

To test the efficacy of constructivist versus exposition-centered course designs, we focused on the design of class sessions — More specifically, we compared the results of experiments that documented student performance in courses with at least some active learning versus traditional lecturing, by metaanalyzing

We followed guidelines for best practice in quantitative reviews ( SI Materials and Methods ), and evaluated student performance using two outcome variables: (i) scores on identical or formally equivalent examinations, concept inventories, or other assessments; or (ii) failure rates, usually measured as the percentage of students receiving a D or F grade or withdrawing from the course in question (DFW rate).

 

Does it lower failure rates?

 

Results

 

The overall mean effect size for performance on identical or equivalent examinations, concept inventories, and other assess-ments was a weighted standardized mean difference of 0.47 (Z = 9.781, P << 0.001) — The overall mean effect size for failure rate was an odds ratio of 1.95 (Z = 10.4, P << Average failure rates were 21.8% under active learning but 33.8% under tradi-tional lecturing — 1 and S1 ).

 

Significance

 

The analysis supports theory claiming that calls to in-crease the number of students receiving STEM degrees could be answered, at least in part, by abandoning traditional lecturing in favor of active learning.

 

Author contributions: S.F. and M.P.W. designed research; S.F., M.M., M.K.S., N.O., H.J., and M.P.W. performed research; S.F. and S.L.E. analyzed data; and S.F., S.L.E., M.M., M.K.S., N.O., H.J., and M.P.W. wrote the paper.

 

The authors declare no conflict of interest.

 

*This Direct Submission article had a prearranged editor.

 

Freely available online through the PNAS open access option.

 

See Commentary on page 8319.

 

1 E-mail: [email protected].

 

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10. 1073/pnas.1319030111/-/DCSupplemental .

 

8410–8415 | PNAS | June 10, 2014 | 111 | 23 www.pnas.org/cgi/doi/10.1073/pnas.1319030111

SEE COMMENTARY

 

 

 

 

 

 

 

 

 

 

 

 

 

The mean failure rates under each classroom type (21.8% and 33.8%) are shown by dashed vertical lines.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

from https://www.pnas.org by 129.19.63.117 on March 21, 2024 from IP address 129.19.63.117.

2A; Q = 910.537, df = 7, P = 2B; Q = 11.73, df = 6, P = 2, Figs. S2 and S3 , and Tables S1A and S2A Thus, the data indicate that active learning increases student performance across the STEM disciplines.

 

3A and Table S1B ; Q = 10.731, df = 1, P << This explanation is consistent with previous research

indicating that active learning has a greater impact on student mastery of higher- versus lower-level cognitive skills (6 – Most concept inventories also undergo testing for validity, reliability, and readability.

 

3B and Table S1C ; Q = 6.726, df = 2, P = 0.035; S4 Effect sizes were sta-tistically significant for all three categories of class size, how-ever, indicating that active learning benefitted students in medium (51 – 110 students) or large ( > 110 students) class sizes as well.

 

When we metaanalyzed the data by course type and course level, we found no statistically significant difference in active learning ’ s effect size when comparing (i) courses for majors versus nonmajors (Q = 0.045, df = 1, P = 0.883; Table S1D ), or (ii) introductory versus upper-division courses (Q = 0.046, df = 1, P = 0.829; Tables S1E and S2D ).

 

 

 

 

 

 

 

 

 

 

 

PSYCHOLOGICAL AND	COGNITIVE SCIENCES

 

 

Downloaded

 

Numbers below data points indicate the number of independent studies; horizontal lines are 95% confidence intervals.

 

Freeman et al. PNAS | June 10, 2014  | 111 | 23 | 8411

 

 

 

 

 

 

 

 

 

Numbers below data points indicate the number of independent studies; horizontal lines are 95% confidence intervals.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

129.19.63.117 on March 21, 2024 from IP address 129.19.63.117.

We calculated fail-safe numbers indicating how many missing studies with an effect size of 0 would have to be pub-lished to reduce the overall effect sizes of 0.47 for examination performance and 1.95 for failure rate to preset levels that would be considered small or moderate — The fail-safe numbers were high: 114 studies on exam-ination performance and 438 studies on failure rate ( SI Materials and Methods Analyses of funnel plots ( S5 ) also support a lack of publication bias ( SI Materials and Methods ).

 

We created four categories to characterize the quality of the controls over student equivalence in the active learning versus lecture treatments ( SI Materials and Methods ), and found that there was no heterogeneity based on methodological quality (Q = 2.097, df = 3, P = Analyzing variation with respect to controls over instructor identity also produced no evidence of heterogeneity (Q = 0.007, df = 1, P = Thus, the overall effect size for examination data appears robust to variation in the methodological rigor of published studies.

Discussion

 

The heterogeneity analyses indicate that (i) these increases in achievement hold across all of the STEM disciplines and occur in all class sizes, course types, and course levels; and (ii) active learning is particularly beneficial in small classes and at increasing performance on concept inventories.

 

Thus, our results are consistent with previous work by other investigators.

 

For example, because struggling students are more likely to drop courses than high-achieving students, the reductions in withdrawal rates under active learn-ing that are documented here should depress average scores on assessments — meaning that the effect size of 0.47 for examina-tion and concept inventory scores may underestimate active learning ’ s actual impact in the studies performed to date ( SI Materials and Methods In contrast, it is not clear whether effect It is an open question whether student performance would in-crease as much if all faculty were required to implement active learning approaches.

 

Assuming that other instructors implement active learning and achieve the average effect size documented here, what would

 

 

Downloaded from https://www.pnas.org by

Table 1. Comparing effect sizes estimated from well-controlled versus less-well-controlled studies

 

					95% confidence interval
Type of control	n	Hedges’s g	SE	Lower limit		Upper limit
For student equivalence
Quasirandom—no data on student equivalence	39	0.467	0.102	0.268		0.666
Quasirandom—no statistical difference in prescores	51	0.534	0.089	0.359		0.709
on assessment used for effect size
Quasirandom—no statistical difference on metrics	51	0.362	0.092	0.181		0.542
of academic ability/preparedness
Randomized assignment or crossover design	16	0.514	0.098	0.322		0.706
For instructor equivalence
No data, or different instructors	59	0.472	0.081	0.313		0.631
Identical instructor, randomized assignment,	99	0.492	0.071	0.347		0.580
or ≥3 instructors in each treatment

 

8412  | www.pnas.org/cgi/doi/10.1073/pnas.1319030111 Freeman et al.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

103 Paragraph 103 changes Downloaded from https://www.pnas.org by 129.19.63.117 on March 21, 2024 from IP address 129.19.63.117. 0 0

 

 

a shift of 0.47 SDs in examination and concept inventory scores mean to their students?

 

i) Students performing in the 50th percentile of a class based on traditional lecturing would, under active learning, move to the 68th percentile of that class (13) — meaning that instead of scoring better than 50% of the students in the class, the same individual taught with active learning would score better than 68% of the students being lectured to.

 

ii) According to an analysis of examination scores in three intro-ductory STEM courses ( SI Materials and Methods On a letter-based system, medians in the courses analyzed would rise from a B − to a B or from a B to a B + .

 

A recent review of educational interventions in the K – Thus, the effect size of active learning at the undergraduate level appears greater than the effect sizes of educational innovations in the K – 12 setting, where effect sizes of 0.20 or even smaller may be considered of policy interest (14).

There are also at least two ways to view an odds ratio of 1.95 for the risk of failing a STEM course:

 

i) If the experiments analyzed here had been conducted as ran-domized controlled trials of medical interventions, they may have been stopped for benefit — Both criteria were met for failure rates in the education studies we analyzed: The average relative risk was 0.64 and the P value on the overall odds ratio was << Any analogy with biomedical trials is qual-ified, however, by the lack of randomized designs in studies that included data on failure rates.

 

Based on conservative assumptions ( SI Materials and Methods ), this translates into over US$3,500,000 in saved tuition If active learning were implemented widely, the total tuition dollars saved would be orders of magnitude larger, given that there were 21 million students enrolled in US colleges and universities alone in 2010, and that about a third of these students intended to major in STEM fields as entering freshmen (17, 18).

 

For example, the 2012 President ’ s Council of Advisors on Science and Technology report calls for an additional one million STEM majors in the United States in the next decade — requiring a 33% increase

 

 

from the current annual total — According to a recent cohort study from the National Center for Education Statistics (19), there are gaps of 0.5 and 0.4 in the STEM-course grade point averages (GPAs) of first-year bachelor ’ s and associate ’ A 0.3 “ bump ” in average grades with active learning would get the “ leavers ” close to the current perfor-mance level of “ persisters. ” Other analyses of students who leave STEM majors indicate that increased passing rates, higher grades, and increased engagement in courses all play a positive role in re-tention (20 – 22).

 

Instead, it may be more pro-ductive to focus on what we call “ second-generation research ” Although the time devoted to active learning was highly variable in the studies analyzed here, ranging from just 10 – 15% of class time being devoted to clicker questions to lecture-free “ studio ” environments, we were not able to evaluate the relationship between the intensity (or type) of active learning and student performance, due to lack of data ( SI Materials and Methods ).

 

As research continues, we predict that course designs inspired by second-generation studies will result in additional gains in student achievement, especially when the types of active learning interventions analyzed here — which focused solely on in-class innovations — are combined with required exercises that are completed outside of formal class sessions (26).

 

Although traditional lecturing has dominated undergraduate instruction for most of a millen-nium and continues to have strong advocates (29), current evi-dence suggests that a constructivist “ ask, don ’ t tell ” approach may lead to strong increases in student performance — amplifying recent calls from policy makers and researchers to support faculty who are transforming their undergraduate STEM courses (5, 30).

 

Materials and Methods

 

We then coded elements in the responses to create the following con-sensus definition:

 

Active learning engages students in the process of learning through activities and/or discussion in class, as opposed to passively listening

SEE COMMENTARY

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PSYCHOLOGICAL AND	COGNITIVE SCIENCES

 

Freeman et al. PNAS | June 10, 2014  | 111 | 23 | 8413

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Downloaded from https://www.pnas.org by 129.19.63.117 on March 21, 2024 from IP address 129.19.63.117.

31, p. iii).

 

Under this definition, student activity was assumed to be limited to taking notes and/or asking occasional and unprompted questions of the instructor.

 

We used four approaches (35) to find papers for consideration: hand-searching every issue in 55 STEM education journals from June 1, 1998 to January 1, 2010 ( Table S3 ), searching seven online databases using an array of terms, mining reviews and bibliographies ( SI Materials and Methods ), and “snowballing” from references in papers admitted to the study ( SI Materials and Methods We had no starting time limit for admission to the study; the ending cutoff for consideration was completion or publication before January 1, 2010.

 

We coded studies that (i) contrasted traditional lecturing with any active learning intervention, with total class time devoted to each approach not differing by more than 30 min/wk; (ii) occurred in the context of a regularly scheduled course for undergraduates; (iii) were largely or solely limited to changes in the conduct of the regularly scheduled class or recitation sessions; (iv) involved a course in astronomy, biology, chemistry, computer science, engineering, geology, mathematics, natural resources or environmental science, nutrition or food science, physics, psychology, or statistics; and (v) included data on some aspect of student academic performance.

 

Note that criterion i yielded papers representing a wide array of active learning activities, including vaguely defined “cooperative group activities in class,” in-class worksheets, clickers, problem-based learning (PBL), and studio classrooms, with intensities ranging from 10% to 100% of class time ( SI Materials and Methods Thus, this study’s intent was to evaluate the average effect of any active learning type and intensity contrasted with traditional lecturing.

 

The literature search yielded 642 papers that appeared to meet these five criteria and were subsequently coded by at least one of the authors.

 

The 244 “easy rejects” were excluded from the study after the initial coder (S.F.) determined that they clearly did not meet one or more of the five criteria for admission; a post hoc analysis suggested that the easy rejects were justified ( SI Materials and Methods ).

 

The two coders met to review each of the remaining 398 papers and reach consensus (37, 38) on

 

i) The five criteria listed above for admission to the study;

 

ii) Examination equivalence—meaning that the assessment given to stu-dents in the lecturing and active learning treatment groups had to be identical, equivalent as judged by at least one third-party observer recruited by the authors of the study in question but blind to the hy-pothesis being tested, or comprising questions drawn at random from a common test bank;

 

iii) Student equivalence—specifically whether the experiment was based on randomization or quasirandomization among treatments and, if quasir-andom, whether students in the lecture and active learning treatments were statistically indistinguishable in terms of (a) prior general academic performance (usually measured by college GPA at the time of entering the course, Scholastic Aptitude Test, or American College Testing scores), or (b) pretests directly relevant to the topic in question;

 

iv) Instructor equivalence—meaning whether the instructors in the lecture and active learning treatments were identical, randomly assigned, or consisted of a group of three or more in each treatment; and

 

v) Data that could be used for computing an effect size.

 

If the data reported were from iterations of the same course at the same institution, we combined data recorded for more than

We also combined data from multiple outcomes from the same study (e.g., a series of equivalent midterm examinations) ( SI Materials and Methods Coders also extracted data on class size, course type, course level, and type of active learning, when available.

 

The data analyzed and references to the corresponding papers are archived in Table S4 .

 

Before analyzing the data, we inspected the distribution of class sizes in the study and binned this variable as small, medium, and large ( SI Materials and Methods We also used established protocols (38, 39) to combine data from multiple treatments/controls and/or data from multiple outcomes, and thus produce a single pairwise comparison from each in-dependent course and student population in the study ( SI Materials and Methods ).

 

Note that although the cluster correction has a large influence on the variance for each study, it does not influence the effect size point estimate substantially.

 

All reported P values are two-tailed, unless noted.

 

The random effect size model was appropriate because conditions that could affect learning gains varied among studies in the analysis, including the (i) type (e.g., PBL versus clickers), intensity (percentage of class time devoted to constructivist activities), and implementation (e.g., graded or ungraded) of active learning; (ii) student population; (iii) course level and discipline; and (iv) type, cognitive level, and timing—relative to the active learning exercise— of examinations or other assessments.

 

For ease of interpretation, we then converted log-odds values to odds ratio, risk ratio, or relative risk (49).

 

To evaluate the influence of publication bias on the results, we assessed funnel plots visually (50) and statistically (51), applied Duval and Tweedie’s trim and fill method (51), and calculated fail-safe Ns (45).

 

The average odds ratio for these 11 studies was 1.97 ± 0.36 (SE)—almost exactly the effect size calculated from the entire dataset.

 

Although we did not metaanalyze the data using “vote-counting” approaches, it is informative to note that of the studies reporting statistical tests of examination score data, 94 reported significant gains under active learning whereas only 41 did not ( Table S4A ).

 

Additional results from the analyses on publication bias are reported in Supporting Information.

 

8414  | www.pnas.org/cgi/doi/10.1073/pnas.1319030111 Freeman et al.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Downloaded from https://www.pnas.org by 129.19.63.117 on March 21, 2024 from IP address 129.19.63.117.

 

 

We thank Roddy Theobald for advice on interpreting odds ratios; the many authors who provided missing data upon request ( SI Materials and Methods ); Colleen Craig, Daryl Pedigo, and Deborah Wiegand for supplying information on examination score standard deviations and

 

1. A History of the University in Europe, ed de Ridder-Symoens H (Cambridge Univ Press, Cambridge, UK), Vol II, pp 565–620.

 

2. Piaget J (1926) The Language and Thought of the Child (Harcourt Brace, New York).

 

3. Vygotsky LS (1978) Mind in Society (Harvard Univ Press, Cambridge, MA).

 

4. Science 304(5670):521–522.

 

5. PCAST STEM Undergraduate Working Group (2012) Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engi-neering, and Mathematics, eds Gates SJ, Jr, Handelsman J, Lepage GP, Mirkin C (Office of the President, Washington).

 

6. J Res Sci Teach 20(7): 629–637.

 

7. Ann Biomed Eng 35(8): 1312–1323.

 

8. J. Eng Ed 98(4).

 

9. CBE Life Sci Educ 10(1):64–73.

 

10. CBE Life Sci Educ 9(4): 435–440.

 

11. Science 331(6022):1269–1270.

 

12. Rev Educ Res 69(1):21–51.

 

13. J Chem Educ 77(1):116–119.

 

14. (2012) Translating the Statistical Representation of the Effects of Educational Interventions into Readily Interpretable Forms (US Department of Edu-cation, Washington).

 

15. JAMA 294(17):2203–2209.

 

16. Clin Trials 3(6):513–521.

 

17. National Center for Education Statistics (2012) Digest of Education Statistics (US De-partment of Education, Washington).

 

18. National Science Board (2010) Science and Engineering Indicators 2010 (National Science Foundation, Arlington, VA).

 

19. National Center for Education Statistics (2012) STEM in Postsecondary Education (US Department of Education, Washington).

 

20. Seymour E, Hewitt NM (1997) Talking About Leaving: Why Undergraduates Leave the Sciences (Westview Press, Boulder, CO).

 

21. (2002) Final Report of the Women’s Experiences in College Engi-neering (WECE) Project (Goodman Research Group, Cambridge, MA).

 

22. J Coll Sci Teach 42(5):36–41.

 

23. Educ Psychol Rev 24(4):569–608.

 

24. Psych Sci Publ Int 14(1):4–58.

 

25. Evol Ed Outreach 6:1–11.

 

26. CBE Life Sci Educ 10(2):175–186.

 

27. Am J Phys 74(2):118–122.

 

 

grading thresholds; Kelly Puzio and an anonymous reviewer for advice on analyzing data from quasirandom studies; and Steven Kroiss, Carl Wieman, and William Wood for comments that improved the manuscript. M.S. was supported in part by National Science Foundation Grant 0962805.

 

28. Science 332(6034): 1213–1216.
29. Change 6:31–34.

 

30. J Res Sci Teach 48(8): 952–984.
31. Bonwell CC, Eison JA (1991) Active Learning: Creating Excitement in the Classroom

 

(George Washington Univ, Washington, DC).

 

32. (Jossey-Bass, San Francisco).

 

33. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 73–101.
34. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 103–125.

 

35. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 51–71.

 

36. Lipsey MW, Wilson DB (2001) Practical Meta-Analysis (Sage Publications, Thousand Oaks, CA).

 

37. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 177–203.
38. Available at www. cochrane-handbook.org. Accessed December 14, 2012.

 

39. The Handbook of Systematic Review and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 221–235.
40. J Educ Behav Stat 32(2): 151–179.
41. Stat Med 21(19):2971–2980.

 

42. Available at http://campbellcollaboration.org/lib/project/167/. Accessed December 10, 2013.
43. Puzio K, Colby GT (2013) Cooperative learning and literacy: A meta-analytic review.

 

J Res Ed Effect 6(4):339–360.

 

44. Educ Eval Policy Anal 29:60–87.

 

45. (2006) Comprehensive Meta-Analysis (Biostat, Inc., Englewood, NJ).

 

46. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 38–47.

 

47. The Hand-book of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 295–315.
48. Ecology 80(4):1142–1149.

 

49. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 237–253.

 

50. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 417–433.

 

51. The Handbook of Research Synthesis and Meta-Analysis, eds Cooper H, Hedges LV, Valentine JC (Russell Sage Foundation, New York), pp 435–452.

SEE COMMENTARY

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PSYCHOLOGICAL AND	COGNITIVE SCIENCES

 

Freeman et al. PNAS | June 10, 2014  | 111 | 23 | 8415