A Guest Post by Kevin Lapp: Multiple Choice Exams are (a) good assessment tools, (b) bad assessment tools, (c) all of the above.

Multiple choice testing is a popular assessment format inEvidence courses, more popular in my experience in Evidence than in othercourses. Some professors use it exclusively, while others mix multiple-choicetesting with essay questions on their exams. There’s good reason for usingmultiple choice testing in Evidence courses. For one, the MBE portion of thebar exam contains multiple choice Evidence questions, so doing so as part of afinal exam helps prepare students for the bar exam format. In addition,multiple choice testing has been around, and is widely accepted as a credibleformat to assess student knowledge. Evidence is also a heavily rule-based classthat lends itself to an assessment format that requires students to identify asingle correct answer. Finally, multiple choice questions allow professors toassess more topics than can be squeezed into an essay question, reducing the chancesthat a student performs well on an exam because he happened to know the issuescovered by the essay questions. 

But there can be a large gap between good multiple choice questionsand bad multiple choice questions. This post is about how those of us who douse multiple choice questions can know if we are doing it in a way that makesfor good assessment. The credibility of our multiple choice questions as soundassessment tools is particularly important given the high stakes testing thatgoes on in so many law school classrooms. When the great bulk, if not theentire portion, of a student’s grade hinges on a single 3 or 4 hour exam, it is our duty totake advantage of the available tools to ensure that our exams function ascredible assessment tools.

A great resource for me as I set about the task of writing myfirst multiple-choice section of an exam was Susan Case & Beth Donahue’sarticle, “Developing High-Quality Multiple-Choice Questions for Assessment in LegalEducation.” (58 J. Legal Educ. 372 (2008)). In it, Case and Donahue note thatmultiple choice questions have several advantages over other formats, includingcontent coverage, grading ease and consistency, and reliability of scores. Butthe reliability of the scores (as a reflection of learning) depends on thequality of the questions. Poorly written questions fail to assess what isintended and fail to encourage the desired learning. The article includes avery helpful discussion of how to draft high-quality multiple choice questions(and answer choices) that I won’t recount here but strongly recommend.

Another indispensable resource forthose who use multiple choice questions on exams was provided to me by myschool. At Loyola, professors receive a detailed report with the results of theobjective portion of a final exam. This report includes means, modes andvariances for the exam as a whole, as well as information on each individualquestion. The information on individual questions includes the percentage ofstudents who selected each possible answer, the performance on the question ofthe top and bottom 27% of the class, and something called the question’s “pointbiserial” (a number which reflects how well a particular question isdiscriminating amongst the students) With this information, a professor can seewhether everyone in the class got #1 correct (fine as an easy warm-up, thoughmeaningless for discriminating amongst students with regard to knowledge),whether the answers selected were distributed evenly amongst the four answerchoices for another question (a sign that the question isn’t working), andwhether those who scored in the top fourth of the class performed better onindividual questions than those who scored in the bottom fourth of the class (asign of a good question because it discriminates between low-performing andhigh-performing students).

Despite doing research and findingthings like the Case and Donahue article ahead of time, I was not confident inmy ability to write a meaningful multiple choice assessment tool the first timeI taught Evidence. Happily, the post-exam report suggested that the greatmajority of my questions were good questions. More importantly, the reportallowed me (with the help of folks who could interpret the data) to get rid ofor try to fix the questions that weren’t credibly assessing learning, making myexam the next time around a more valid and reliable instrument. For those whoseschools don’t provide this kind of data, I’d encourage you to ask for it.

And as the calendar turns to November,and I start drafting some new questions for this semester’s exam, I’d also behappy to hear other suggestions and strategies for creating good multiplechoice questions.