Essay-Grading Software Offers Professors a Break - by John Markoff, New York Times, April 4, 2013
0 General Document comments
0 Sentence and Paragraph comments
0 Image and Video comments
Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the “send” button when you are done and receiving a grade back instantly, your essay scored by a software program.
And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade.
I believe that students benefit greatly from correcting their mistakes, especially in a mathematics classroom. Students receive their initial grade on the assessment, but then are given the opportunity to work through the assessment again to see their mistakes and correct them. Some teachers allow for students to get half credit back on this or others, like me, use this as a homework grade for completion. As for essays, we are most often given the opportunity to turn in a first draft if not a second then third before handing in for the final draft. I see this as the same concept.
EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks.
The new service will bring the educational consortium into a growing conflict over the role of automation in education. Although automated grading systems for multiplechoice and truefalse tests are now widespread, the use of artificial intelligence technology to grade essay answers has not yet received widespread endorsement by educators and has many critics.
Anant Agarwal, an electrical engineer who is president of EdX, predicted that the instantgrading software would be a useful pedagogical tool, enabling students to take tests and write essays over and over and improve the quality of their answers. He said the technology would offer distinct advantages over the traditional classroom system, where students often wait days or weeks for grades.
“There is a huge value in learning with instant feedback,” Dr. Agarwal said. “Students are telling us they learn much better with instant feedback.”
But skeptics say the automated system is no match for live teachers. One longtime critic, Les Perelman, has drawn national attention several times for putting together nonsense essays that have fooled software grading programs into giving high marks. He has also been highly critical of studies that purport to show that the software compares well to human graders.
“My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman, a retired director of writing and a current researcher at M.I.T.
He is among a group of educators who last month began circulating a petition opposing automated assessment software. The group, which calls itself Professionals Against Machine Scoring of Student Essays in HighStakes Assessment, has collected nearly 2,000 signatures, including some from luminaries like Noam Chomsky.
“Let’s face the realities of automatic essay scoring,” the group’s statement reads in part. “Computers cannot ‘read.’ They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others.”
But EdX expects its software to be adopted widely by schools and universities. EdX offers free online classes from Harvard, M.I.T. and the University of California, Berkeley; this fall, it will add classes from Wellesley, Georgetown and the University of Texas. In all, 12 universities participate in EdX, which offers certificates for course completion and has said that it plans to continue to expand next year, including adding international schools.
The EdX assessment tool requires human teachers, or graders, to first grade 100 essays or essay questions. The system then uses a variety of machinelearning techniques to train itself to be able to grade any number of essays or answers automatically and almost instantaneously.
The software will assign a grade depending on the scoring system created by the teacher, whether it is a letter grade or numerical rank. It will also provide general feedback, like telling a student whether an answer was on topic or not.
Dr. Agarwal said he believed that the software was nearing the capability of human grading.
“This is machine learning and there is a long way to go, but it’s good enough and the upside is huge,” he said. “We found that the quality of the grading is similar to the variation you find from instructor to instructor.”
EdX is not the first to use automated assessment technology, which dates to early mainframe computers in the 1960s. There is now a range of companies offering commercial programs to grade written test answers, and four states — Louisiana, North Dakota, Utah and West Virginia — are using some form of the technology in secondary schools. A fifth, Indiana, has experimented with it. In some cases the software is used as a “second reader,” to check the reliability of the human graders.
But the growing influence of the EdX consortium to set standards is likely to give the technology a boost. On Tuesday, Stanford announced that it would work with EdX to develop a joint educational system that will incorporate the automated assessment technology.
Two startups, Coursera and Udacity, recently founded by Stanford faculty members to create “massive open online courses,” or MOOCs, are also committed to automated assessment systems because of the value of instant feedback.
“It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right,” said Daphne Koller, a computer scientist and a founder of Coursera.
Allowing students to resubmit work “until they get it right” is lunacy. I could understand if a student completely missed the point of an assignment or if instructions weren’t clear, or if the nature of the assignment was to elicit feedback and then revise. But, simply letting students resubmit assignments until those assignments resemble something that we think makes it look like they’ve learned something is a disaster waiting to happen.
Last year the Hewlett Foundation, a grantmaking organization set up by one of the HewlettPackard founders and his wife, sponsored two $100,000 prizes aimed at improving software that grades essays and short answers. More than 150 teams entered each category. A winner of one of the Hewlett contests, Vik Paruchuri, was hired by EdX to help design its assessment software.
“One of our focuses is to help kids learn how to think critically,” said Victor Vuchic, a program officer at the Hewlett Foundation. “It’s probably impossible to do that with multiplechoice tests. The challenge is that this requires human graders, and so they cost a lot more and they take a lot more time.”
Mark D. Shermis, a professor at the University of Akron in Ohio, supervised the Hewlett Foundation’s contest on automated essay scoring and wrote a paper about the experiment. In his view, the technology — though imperfect — has a place in educational settings.
With increasingly large classes, it is impossible for most teachers to give students meaningful feedback on writing assignments, he said. Plus, he noted, critics of the technology have tended to come from the nation’s best universities, where the level of pedagogy is much better than at most schools.
I absolutely agree that meaningful feedback is one of the main, if not the only, ways for students to grow as writers. Especially in a college course, there could be up to 300 students in an entry level English course; there is no way that a teacher can leave meaning critiques on every students paper even with graduate assistances. I think this method of grading is amazing and I can’t wait to see how this impacts teachers in the high school level.
“Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could,” Dr. Shermis said. “There seems to be a lack of appreciation of what is actually going on in the real world.”
General Document Comments 0
Like so much other technology in education, this is initially being received hesitantly and with a lot of criticism. But I think it will be developed to a point where it will be a useful tool and provide valuable support to instructors and students.
Spelling and grammar checks in microsoft word are not always correct, but they are often enough to be very useful and improve the standard of writing. In the same way I think this technology may help students prior to submitting essays.
As far as replacing a human grader goes, that depends on how knowledgeable the human is! The technology may progress to a point where it does just as well as many graders, but may never replace a well-qualified English teacher.
With the increasing numbers of students and pressure on teachers, I think this sort of technology is inevitable and will form part of a system that will become predominantly automated.
Like the discussion on translating earlier in the semester, I think this technology is worth researching as it will definitely provide benefits, but it may never replace a qualified individual.
As an upcoming English teacher, I may be hesitant to implement an auto-grading essay system in my classroom. With our current technology, a machine that grades a paper automatically will have many flaws when compared to the traditional method.
One of the biggest flaws that I worry about is the lack of personality that a computer has when compared to a human. Many teachers teach what they were taught, so not every essay will be physically scored in a similar manner. Although this sounds like a bad thing, it is actually good because the learner can learn new concepts from different perspectives. We just can’t get that from a computer programmed to grade an essay with a one-tracked mind.
Another problem is the inevitable use of cheating techniques. When a teacher hand grades an essay, it is harder for a student to cheat, since the essay is directly being looked upon. With an automated essay scoring system, students will eventually find out what the software looks for in an essay. If a teacher only lets the computer check the assignment, then a student can pass an essay that he/she barely took the time to do.
I understand the benefits that a automated grading system for essays could bring. This could be quick feedback and lack of bias. Despite this, having a computer grade my essay would make me uncomfortable as a student. The article admitted that this software still has a long way to go and honestly I do not see a computer being more accurate than a trained professional who understands the context and the assignment. However, I do think that it would be beneficial to use both. The teacher could use the software to check things that the teacher could have missed or to gain different perspective. The teacher could also let the students have one re-do by using the software which could help the students better understand their grammar mistakes and overall writing style. So, I do see a place for this software, but I do not think the teacher should be totally left out of grading essays. -Maggie Trimble
I can see where this automatic grading could be useful when grading multiple choice and matching answers. However, I do not see the benefit of having this software grading short answers and essay.
I might be one of the skeptics, but in my mind there is no way for a machine to take words and understand the exact context in which the writer is meaning to use them in. I feel like if this software was to be used in schools then children would eventually be graded unfairly which would result in a whole other mess in itself having to deal with upset students and parents.
I cannot say that I support this method of grading. I understand the want for instant feedback on a student’s part, but I can’t see how the feedback will always be relevant. You can have as many lines of code as you want to try to interpret all meanings, but if there is even one instance where the machine interprets incorrectly, the system is flawed in my opinion.
It may take longer, but when a professor grades a paper, they can be approached with questions. You can ask the professor why they thought a certain way about your answer, or back your answer up with evidence. You don’t have that option with a machine grading an essay.
The fact is when it comes to essays and written word, the grading should come from a qualified teacher that actually taught you the material. Written word should be analyzed by someone that can feel emotion, or simply interpret on multiple levels. The concept of the software is nice, but I think the instant grades should be continue to apply only with multiple choice or true of false questions. Anything that can be interpreted in multiple ways should be graded by a human and not a machine that supposedly thinks like said human.
If the purpose is merely to give grades then there are obvious advantages. Unfortunately I think there are some serious problems with the way we expect students to write essays over and over and 95% of them are just for a grade with no real audience. Academic writing has became so formulated that it is not particularly hard to put together all the “required” elements to get a good grade without really putting any original ideas on paper. And really there is little incentive to come up with an original argument if your audience is just a computer.
One question arises though, as teachers instruct on writing in different manners, the ‘grader’ (I suppose) would grade the same way, each time. So, is there a way that the teacher can alter some of the ‘grading’ or is there a ‘rubric’ so to speak, against which the teacher instructs.
I say this only when I see frustrated students, as they learn to write from teacher ‘A’, then the following year teacher ‘B’ is focusing on other elements of the writing, and they feel they are starting all over again. With this kind of tool, it seems that its ‘one direction’ or ‘method’.
This technology has an amazing upside. With anything new there will be skeptics but the positive aspects outweigh the negative implications. More and more people are going to college and online colleges are expanding more and more. The number of people attending college have increased dramatically and oftentimes students have to wait weeks to receive grades on essays. This dramatically affects instruction as teachers are forced to give multiple choice questions or short answer questions. This dramatically affects the way that professors evaluate students understanding of the content as it difficult for them to assess true mastery of the content via multiple choice questions and short answers. This would dramatically improve pedagogy in the classroom and increase effective learning that would drive instruction. There are some drawbacks to using an automated system as there would be some papers that may be incorrectly graded but the positive implications of this new technology dramatically outweighs all these possible negative things that may occur and has the ability to take education into a new direction
icons on the left to see existing comments.
0 archived comments