Friday, November 23, 2007

The Fairness versus Usefulness Tradeoff

Game designers love tradeoffs. Tradeoffs are one kind of "interesting decision". Life is full of all kinds of tradeoffs (it has in fact been argued that life is simply a massively-multiplayer game... with extremely realistic graphics). Teaching has its fair share as well.

Here's one that I'm discovering: the more useful an experience that some exercise (an in-class presentation, homework problem or project) is, the more subjective the grading is.

It's very easy to create a multiple-choice exam where each question has exactly one, clearly correct answer. The grading is easy and the grades themselves are inherently fair. But students don't actually learn anything by taking the exam, they merely show you what they've memorized. The same is true for homework questions that are direct enough to have exactly correct answers.

By contrast, essay questions, open-ended student projects and creative exercises all lend themselves to both student and teacher interpretation. Even if you think you know what you're looking for in order to grade objectively, some students will surprise you with unexpected answers and you'll have to revise your grading methods. Sometimes a student answer will, on reflection, be better than your own. Sometimes you will simply not understand what a student is saying, and they will be completely correct and it's your own fault for misinterpreting their answer (you can weasel out of this by saying "if I didn't understand it then it was poorly written" but I would consider that the height of arrogance in most cases).

And yet, these subjectively-graded exercises are the most valuable because they force the students to actually design something.

In terms of teacher evaluations, this means that any class where I score well on "relevance to the real world" is probably also a class where I score poorly on "fairness of grading"...


Anonymous said...

You really need a balance an exam with multiple choice questions, essay questions, etc.

You can write really good multiple choice questions that can easily require the student to apply their knowledge. A well written multiple choice question isn't exactly an easy task.

If an exam is purely essay, the instructor/professor really needs to stand up and prepare the students much more (for that particular exam, not just lecturing). My luck with essay type exams has been not-so-great. A few of my instructors/professors have lead my classes into the exam totally blind.

Ian Schreiber said...

I hate to ask, but I don't suppose you could provide an example of a really good multiple choice question?

In all my time in grade school and college, I don't think I ever took an exam where I said to myself "wow, that was a great multiple choice question, it really got me thinking!"

You're right though, just because I've never seen it done doesn't mean it's impossible.

Anonymous said...

I've saved some of my exams. I'll take a look through them to try and find some examples of good and bad.

I don't think I've ever said "wow, that was a great MC question", either. I personally don't enjoy exams, so I tend to forget them after I've taken it. But, any Science, or Computer Science class, can pump out some pretty good MC questions.

Essentially, a good MC question to me is where you MUST think through each option, because each option is a component of the solution. But, there is always a clear RIGHT answer.

And, a good MC question doesn't waste the last answer as, "Obviously, don't pick me"

An even better/challenging form of a MC question is to create a question where you fill in more than one concept for the question.

A blah blah _____ blah blah blah _____, does blah blah what, if _____ blah.

The solution isn't necessarily 'memorized' if the question is formatted in a way that makes the reader 'understand' why it is what it is. The student MUST understand, or the entire question is missed.

I'll be back around tomorrow with some examples.